Educational Communications 


Presented Online during The Annual Convention of the 
Association for Educational Communications and Technology 


Editors 

Michael Simonson, Ph.D. 
Fischler College of Education 
Nova Southeastern University 
Davie, FL 


Deborah Seepersaud, Ed.D. 
Extended Learning 
Academic Affairs 

Barry University 

Miami Shores, FL 


2020 Annual Proceedings — Volumes | & 2 


Volume 1: Selected Research and Development Papers 
And 
Volume 2: Selected Papers 
On the Practice of Educational Communications and Technology 


Presented Online during 
The Annual Convention of the Association for Educational Communications and Technology 
2020 


Editors 
Michael Simonson, Ph.D. 
Fischler College of Education and School of Criminal Justice 
Nova Southeastern University 
Davie, FL 


Deborah Seepersaud, Ed.D. 
Extended Learning 
Academic A ffairs 
Barry University 
Miami Shores, FL 


Preface 


For the forty third time, the Association for Educational Communications and Technology (AECT) is 
sponsoring the publication of these Proceedings. Papers published in this volume were presented line 
during the annual AECT Convention. A limited quantity of these Proceedings were printed and sold in 
both hardcopy and electronic versions. Volumes | and 2 are available through the Educational Resources 
Clearinghouse (ERIC) System. Proceedings volumes are available to members at AECT.org. 


Proceedings copies are also available at: 


http://www.tresystems.com/proceedings/ 


The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Selected Research and Development Papers 


Training, Planning, Designing: Purposeful Professional Development 


Dr. Gary L. Ackerman 
Greenfield Community College 
1 College Drive 
Greenfield, MA (USA) 
teacher professional development, technology leadership 


One purpose of school has always been to help students become competent readers, 
writers, reasoners, and calculators. Implicit in this is that teachers will be prepared to incorporate 
common information technologies into their lessons. For many generations of students in public 
schools, information technologies used for print and the written word dominated schools, but the 
arrival of desktop computers changed that. 

The digital tools installed in schools in recent decades are the latest of the long series of 
electronic media that were developed in the 20" century. Despite promising rhetoric from 
advocates, efforts to use radio, movies, and television in education largely failed (Cuban, 1985) 
and they disappeared from classrooms or were used for marginal purposes. Advocates for 
computer-based instruction appear to have reversed that pattern as those devices and media have 
become important tools for all students 

Researchers associated with the Apple Classrooms of Tomorrow (ACOT) project, one the 
early initiatives to study computers in schools, found teachers will use computers if they are 
taught to use them and if they are supported in learning to teach with them (Sandholtz, Ringstaff, 
& Dwyer, 1997; Schofield, 1995). Following that finding, school and technology leaders have 
implemented professional develop to support teachers’ learning about all aspects of technology. 

Organizing and presenting effective professional development continues to be a problem 
faced by practitioners, leaders, and scholars (Jin, Li, Meirink, and der Want, & Admiraal, 2019; 
Merchie, Tyytens, Geert, & Vanderlinde, 2018). The rapid evolution of technology (de Reuver, 
Sorensen, & Basole, 2018), the effects of technology on pedagogy (Gordy, Jones, & Bailey, 
2018), the lack of a dominant theory of educational technology (Hew, Lan, Tang, Jia, and Lo, 
2019), and a variety of teachers’ personal skills and beliefs (Joo, Park, & Lim, 2018) are all 
factors that have been found to affect teachers’ learning about computers and technology. 

Meeting the professional development needs of teachers can be particularity challenging 
for leaders in rural schools that tend to be smaller and have fewer resources that can be dedicated 
to professional technology staff than suburban and urban schools. This paper describes the 
efforts of one rural school district to improve the professional development opportunities 
available to teachers in the schools they lead. 


Purpose 


A curriculum coordinator summarized the current situation regarding her district’s 
professional development related to information technology for teachers, “we offer technology 
workshops and different people show up expecting different things.” She continued “We need to 
give everyone better information so principals and teachers can make better decisions... so we 
[avoid] frustrating those faculty who make the effort to attend workshops, but [then] leave 
feeling it was a waste of their time.” In order to address this situation, the curriculum coordinator 
initiated a project to design a comprehensive plan for supporting teachers’ learning about 


technology and its role in the classroom. The author was retained as an external consultant to 
support this project. Several themes emerged during the work; eventually, these were sufficiently 
well developed that they were used to define a typology. This paper contains a description of the 
project, the typology that emerged, and the some of the observations of the leaders who 
participated. 


Setting 


This project involved the leaders of a school district in the northeast United States. The 
district comprises six schools, one secondary school (enrolling about 480 students in grades 7- 
12) which is located near the geography center of the school district and five small schools 
enrolling students in grades PK-6 dispersed throughout the towns of the district. 

A team, led by the curriculum coordinator and comprising the principals from all of the 
elementary schools and the assistant principal at the 7-12 school, are largely responsible for the 
instructional leadership decisions in the district. One member of the leadership team captured the 
consensus of the team when she said, “Technology is in our vision statements and the 
school board is supportive... but sometimes we just don’t have the needed experience... my 
background is literacy, I can help our teachers help struggling readers, but technology is not my 
field.” Pointing at another, she continued, “he can do the same for science, and I trust his 
judgement, but who here can we trust when we talk about technology?” 

They described their existing technology support system and professional development as 
“ad hoc” to capture the lack of consistency and direction. In the past, the district had retained a 
technology coordinator who was responsible for both managing the infrastructure and organizing 
and presenting professional development in educational technology. The curriculum coordinator 
described the failed search to fill that position, “We realized [our best candidate] was really good 
at technology but had no education background... so we left the position open.” The leadership 
team concurred the technicians that remined were keeping the technology reliable and robust, but 
“After hiring consultants to do some technology workshops, we decided—as educators—we 
needed to take a more active role leading our work with educational technology,” explained the 
curriculum coordinator. 


Data collection 


Following the recommendation of the author, the leadership team decided to focus their 
work on the question “What can we do to improve our practices related to professional 
development specific to technology?” and to modify Delphi research methods (Delbeq, Van De 
Ven, & Gustafson, 1975) for this project. Similar methodology methodology has been used to 
address planning and procedural problems in situations in which new strategies are deemed 
necessary (Malekpour, de Haan, & Brown, 2016). This project found a panel of experts 
participating in focus group interviews to discuss and give feedback on a series of documents 
prepared by the leadership team (see figure 1). 


Leadership Team Panel of Experts 


What we ; 
did Discuss/ 


Feedback 


Recording & 
Annotations 


Plan #1 Discuss/ 
Feedback 


Recording & 
Annotations 


Typology 


Figure 1. Illustration of the modified Delphi process 


The leadership team began by composing a collection of two-page documents in which 
they identified the professional development that had been held in their building in the previous 
year along with a reflection on how successful or effective they found the efforts to be. That 
collection was forwarded to a panel of experts who discussed the summaries and provided 
feedback in a focus group interview held via video conference 

The annotated documents and audio recordings of the focus group interviews was used 
by the leadership team to compose a draft of a professional development plan for the following 
year. That plan was discussed by the same panel of experts who also provided feedback to the 
leadership team. Audio recordings and the annotated documents from the second focus group 
were incorporated into a second draft of the professional development plan. 

Digital versions of each document created during the process along with transcripts of the 
focus group and the leadership team meetings at which the professional development plans were 
drafted were coded in a multistep process. First, the words that appeared most frequently in the 
documents were used as major categories to code the data (Savin-Baker & Major, 2013). The 
author and one other researcher negotiated the results of the first coding and reduced the codes to 
six and coded the data a second time in a manner consistent with the constant comparative 
method (Glaser, 1965). When coding the data for the second time, both researchers identified 
three themes that dominated the documents. In the final negotiation of the codes, the researchers 
concurred that “focus on technology,” “source of expertise,” and “the role of the students” were 
the dominant themes. The themes were presented to the leadership team who concurred with the 
definitions and define the typology. 


Dimensions of Professional Development 


Because the three themes appeared to describe independent (although connected) factors 
that varied in opposite ways, they were interpreted as dimensions which can be defined along 
continua. Benchmarks along the continua were defined so that different values of each could be 
useful in predicting the nature of the professional development activities offered to teachers. 


Focus on Technology 


While it may seem unnecessary to recognize professional development to improve 
teachers’ use of technology should have a focus on technology, the participants in this project 
developed a sophisticated understanding of technology during the project. Technology was 
differentiated into that which is the same regardless of the role of the teacher (labeled 
decontextualized technology) and that which depends on the situation in which it is used (labeled 
contextualized technology). These two types of technology were placed on the extremes of the 
Focus on Technology dimension (see figure 2). 


Focus on Technology 


Decontextualized Contextualized 
Important for all Narrow groups 


Figure 2. Focus on Technology dimension 


Correlated with contexts in which the technology is used in the definition of the Focus on 
Technology dimension was the size of the population that needs to learn about. The assistant 
principal from the secondary school differentiated the two extremes. “There are some things 
everyone needs to know,” he suggested, “our middle school art teacher does the same things to 
take attendance in [the SIS] as the high school chemistry teacher.” This was contrasted with “the 
fancy graphing tools our calculus teacher uses.” It was reasoned everyone could attend the same 
professional development on the SIS, but only one or two teachers would need to learn about the 
graphing tool. 

The team decided it was not necessary to define benchmarks along the continuum 
representing the Focus on Technology dimension. They recognized the hardware and software 
that focused the professional development would vary in the contexts in which it was relevant 
and the groups that deemed it useful could make those decisions for themselves. 


Source of Expertise 


The professional development planned by the leadership team supports teachers as they 
become more competent in accomplishing curricular goals as well as more competent users of 
technology. The leadership team recognized three types of expertise are necessary to accomplish 
these two goals, and those three define the Source of Expertise dimension (see figure 3). 


Source of Expertise 


Technician Technology Teacher 


specialist 


Figure 3. Source of Expertise dimension 


At one extreme on the dimension, the expertise comes from a technician whose 
“knowledge is dispensed with little concern for who is using it.” An example described by the 
leadership team was the representative from the publishers of their new student information who 
arrived to demonstrate the how to use the graphic user interface to enter data and generate 
reports. At the other extreme is the expertise of teachers who understand teaching and learning 
independent of the technology. Their expertise is in the nature of experiences necessary for 
students to learn the curriculum and methods for delivering those experiences. 

Between the technician and the teachers, there are technology specialists. These 
individuals tend to be teachers who develop technology expertise. Placed between technicians 
and teachers, these individuals serve as a conduit for communication between the two sources of 
expertise. They both recommend technology solutions for teachers and recommend updates and 
reconfigurations of technology to meet teachers’ needs. They help troubleshoot technology with 
teachers and communicate with technicians to seek clarification or to communicate needed 
changes. While technicians are expected to contribute to professional development for specific 
events that would not be repeated for individual teachers, technology specialists are expected to 
form on-going relationships with teachers. 


Role of Students 


The leadership team also recognized students are an important consideration in their 
plans. One of the principals suggested, “this may be the biggest part of what we did, we now 
know how to use students’ experiences to inform professional development for teachers.” The 
Role of Students dimension was defined with three benchmarks (see figure 4). 


Role of Students 


None Prediction on Reaction to 
needs experiences 


Figure 4. Role of Students dimension 


Some professional development can be effective and can be conducted with no 
consideration to the students. These activities are unchanged when there are different populations 
of teachers in the audience. Other professional development is effective only when it is 
conducted with populations of students in mind and decisions are made in a manner that will 


benefit those students. The leadership team did differentiate those professional development 
activities that are conducted without real knowledge of how the technology would be 
experienced by the students and that professional development intended to improve future 
lessons by incorporating students’ and teachers’ experiences into the lessons. 


Typology of Professional Development 


The leadership team defined three types of professional development in terms of the three 
dimensions. For these leaders, these became conceptual artifacts (Bereiter, 2002) as they could 
agree on what would happen during professional development activities labeled as each and the 
team was committed to using the labels. The conceptual artifacts were expressed in a collection 
of planning documents and templates the leadership team developed as the project ended. The 
curriculum coordinator had the support of the rest of the leadership team when she concluded, 
“My office won’t schedule or announcement anything that isn’t classified as one of them.” 


Training 


When professional development activities are focused entirely on how to operate technology and 
the steps were appropriate for any teacher to learn without regard to the students in their classes, 
the team described it as Training. The individual who provided the Training needed no expertise 
in teaching, although teaching expertise did not preclude them from delivering Training (see 
figure 5). 


Focus on Technology 
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Source of Expertise 


Technician 


Role of Students 


Figure 5. Definition of Training 


In addition, the previously mentioned student information system workshops, the 
leadership team suggested learning how to use productivity suites be subject to Training. Further, 
this type of professional development is appropriate for on-boarding new teachers to ensure they 
understand the district’s email, network, human resources, and other systems and procedures. 
Training is not exclusive to operational functions, however; the school librarians were asked to 


provide training in using the library databases and professional staff who work in the school’s 
maker space Trained teachers in using the printers and other devices located in that studio. 


Planning 


The leadership team summarized Planning as “professional development activities in which 
teachers prepare technology-based lessons but they don’t know for sure how they will turn out 
with students.” In terms of the three dimensions, planning is narrowly focused and 
contextualized. Even then they are planning in groups, teachers prepare for the students they will 
teach during a planned period of time. While technology specialists are often present during 
Planning, they usually defer to the expertise of teachers in during this type of professional 
development (see figure 6). 


Focus on Technology 


Contextualized 
Narrow groups 


Source of Expertise 


Teacher 


Role of Students 


Prediction of 
needs 


Figure 6. Planning defined 


The leadership team had intense discussions about the Role of Students in Planning, 
especially with the fact that plans may not actually result in the intended experiences for 
students. The choice of the words “prediction of needs” to define the roles of students resulted 
from negotiation within the team about the multifarious factors that affect student learning and 
the final agreement that “good plans make the outcome likely but doesn’t guarantee them.” 


Design 


The final type of professional development the leadership team is intended to improve the 
effectiveness of the planned activities. Recommendations are made to improve both make the 
technology more efficient and the learning experience more effective. Compared to Planning, 
Design finds a shift towards more broadly applied technology, a greater role for technology 
specialists, and decisions made in response to the experiences of the students (see figure 7). 


Focus on Technology 


Contextualized 
Narrow groups 


Source of Expertise 


Technology Teacher 


specialist 


Role of Students 


Reaction to 
experiences 


Figure 7. Design defined 


In some cases, the technology specialists will be present when the planned activities are 
implemented. In the Design that follows the implementation, the technology specialist interprets 
their observation of the efficiency and effectiveness of the technology and recommends changes 
to improve it for future users. Because future users include both similar groups as well as 
different group that use the same tool for similar purpose, the focus on technology is less 
contextualized than it is for Planning. The recommendations may focus on changes in how the 
technology is configured as well as practiced for how students are directed to use it. 

Perhaps the most innovative aspect of this plan from the leadership team’s perspective 
was the active role that students would play in Design. Their intent was to encourage teachers to 
use this information to inform Design decisions, so the curriculum coordinators clarified she 
would not support Design activities unless they incorporated data regarding students’ experience. 
Further, to encourage Design, Planning proposals were expected to include strategies to gather 
student experiences. 


Discussion 


This project can best be described as an action research project. The participants 
originally intended to simply create a plan for professional development related to technology for 
the following year. It is only after the potential for including the expert panel that the importance 
of including outside participation for creating new strategies was recognized. While the 
leadership team did plan to gather and use the recordings and annotations of the documents for 
their own planning, the decision to code the data was made only after one of the experts on the 
panel asked permission to code the data to gain experience for their forthcoming dissertation that 
would analyze qualitative data. For these reasons, the project can be discussed from the 
perspective of the participants, but generalizations are difficult to justify. 

The members of the leadership team concurred the project had been a valuable learning 
experience, although it had expanded to include more of their time than they originally intended. 


On several instances, the team extended the hours of their regular meetings to accommodate 
work on this project and gave themselves tasks to complete before they next convened. One of 
the principals noted at the end, “I would not have agreed to this knowing how would grow, but I 
see we have a structure for PD into the future rather than just a plan for next year. I definitely 
learned about technology in my school.” 

In discussing the project and the typology, the curriculum coordinator agreed with the 
author that the dimensions and typology were sound and had emerged from the data, but they 
“probably aren’t original. I’m not sure I could point you to them right now, but I bet we could 
find similar ideas with a quick search.” In response, she concluded, “You probably could have 
told us this, but I’m not sure we really would have understood it in a deep manner. Because we 
argued over the feedback, we all share a common understanding. That doesn’t come from being 
told.” For the members of this team, their participation in drafting the plans and negotiating and 
interpreting the feedback from the expert panel interviews was an irreplaceable part of their 
learning. 

The project also represented a shift in operational decisions related to professional 
development. The team intended to schedule training sessions based on the business and data 
requirements of the school district, and to allow and encourage teachers to take a more active 
role in defining, scheduling, and leading their own professional development to plan and design 
curriculum and instruction. Previously, decisions related to professional development offerings 
and attendance were made by the leadership team. Their new understanding led to leadership 
team to decide that, except for Training, many professional development decisions related to 
Planning and Design would be made by teachers and managed by them as well. This change was 
instantiated in the templates and forms the team produces as part of the project as well. 

The final effect on the members of the team was change in their perceived capacity to 
provide effective leadership. At the start of the project, they recognized that (as individuals and 
as a group) they lacked the expertise to provide effective leadership specific to technology. 
“Before this, I saw technology and workshops as a cloud. I wasn’t sure what they did and why, 
and I avoided it whenever I could,” noted one of the participants who continued, “now I at least 
have some questions I can ask to help me understand what teachers need and what workshops 
will entail. That makes be a better leader, as I can facilitate good decision making by my staff.” 


Conclusion 


In conclusion, this project describes how a group of rural school leaders addressed a gap 
in their professional knowledge. While the project did become greater than they originally 
intended, it produced a typology that was not part of the original intent and that increases their 
capacity to provide effective leadership. In reviewing the final report of the work, the curriculum 
coordinator observed, “The typology isn’t what we wanted when we started, but it is what we 
needed.” 
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This paper presents a systematic structured review of recent research that explicitly 
underscores teaching and learning aspects of online feminism. The purpose of this paper is to 
draw attention of higher education professoriate, policy makers and practitioners to social media 
platforms as potential supplementary teaching tools in higher education contexts. 

In this digitally driven world, many people around the world access social media and 
other online platforms, proving that the power of the internet becomes indisputable in modern 
age (Fotopoulou, 2016). Digital media is praised for its potential in fostering global communities 
of feminists who participate in social networks to debate, discuss, share information and call for 
action (Munro, 2013). Women’s voices are amplified in neoliberal digital spaces because social 
media affords interactivity (Pruchniewska, 2018). Digital spaces facilitate the paradigm shift in 
that serving as a new means of organizing campaigns against issues of inequality and social 
injustice (Eudey, 2012). To analyze how feminist pedagogy is practiced in the digital spaces, this 
systematic literature review looks at ten seminal articles that explicitly talk about teaching and 
learning aspects of online feminism. This paper is structured in the following way: first, I discuss 
the rationale of choosing feminist pedagogy as a theoretical framework. Second, I provide 
various scholars’ definitions of feminist pedagogy to enhance our understanding of feminist 
pedagogy. Third, I explain this study’s mode of inquiry and research questions that helped me 
examine the content of the articles. Fourth, I relay findings in relation to the research question. 
Then, I will conclude the paper by making recommendations for key stakeholders in higher 
education. 


Perspective(s) or theoretical framework 


Feminist pedagogy is adopted as a theoretical framework in analyzing the content of the 
articles. Based primarily on feminist theory, this pedagogy includes a set of teaching strategies 
that considers knowledge as socially constructed (Weiler, 1995). The core of feminist pedagogy, 
however, is a teaching method that is geared to empowering the mode of learning that places 
women at the forefront (Luke, 1994). Empowering women in learning is essential in 
contemporary higher education because male styles of learning is rooted in institutional culture, 
making women students feel incapable of succeeding as students (Sengupta & Upton, 2011). 
According to Luke (1994), feminist pedagogy helps women’s identity be more visible in the 
mainstream academic discourse. Second, focusing on feminine scholarship and politicizing 
women’s issues, it offers approaches to educational and consciousness raising practices. Third, it 
acknowledges the diversity of women’s experiences and fosters inclusive content delivery in 
teaching and learning. Before proceeding to the systematic analysis of the articles, I have 
reviewed some articles that define feminist pedagogy to provide a detailed analysis of the 
framework of this study. 
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Definitions of Feminist Pedagogy 


As aterm, feminist pedagogy was coined during the 1980's to describe teaching methods 
that were used by feminist scholars in women's studies programs (Shackelford, 2020). Since that 
time, scholars developed various definitions of feminist pedagogy. Shackelford (2020) describes 
feminist pedagogy as student-centered and the one that emphasizes cooperation in the learning 
and teaching process. Since feminist pedagogy focuses on developing critical thinking, 
Shackelford recommends economists to practice feminist pedagogy for as they can utilize 
dialogic approaches to teaching and to collaborative learning. Grande (2003) defines feminist 
pedagogy as a teaching practice that fosters teaching practices that encourage student-centered 
learning. When applying the values of feminist pedagogy in higher, such as its student-centered 
and cooperative styles, the author recommends analyzing gender through its intersections with 
race, class, and sexuality. Discussing the underrepresentation of women (and men of color) in 
science, Mayberry (1998) describes feminist pedagogy as a teaching practice that helps resist the 
dominant discourse and transform current learning practices by advocating for collaborative 
learning in science. Feminist pedagogy helps educators teach students how to question scientific 
systems and the relationship of those systems to power, oppression, and domination 
(Mayberry, 1998). Maher (1987) defines feminist pedagogy as a teaching practice that is made of 
two sources: liberation pedagogy and feminist theories. As the author argues, these two 
approaches complement each other, and when practiced together, they produce a robust 
pedagogy, which is feminist pedagogy, that can challenge traditional teaching practices in higher 
education. Using such pedagogy in higher education helps promote the role of women in 
education, helping improve teaching both women and minority students (Maher, 1987). 


Modes of Inquiry 


A systematic literature review was chosen as a mode of inquiry for this study with the 
aim of conducting critical content analyses of the studies that are included in this literature 
review. Structured literature review (SLR) is a systematic approach of identifying, collating and 
analyzing a set of studies by defining its boundaries in terms of what will be included and 
excluded (Bisogno, Dumay, Rossi, & Polcini, 2018). Adopting a structured literature review 
methodology and applying the topical category of SLR was deemed as appropriate to directly 
analyze the articles in their relation to research questions. 

This review did not apply exclusionary criteria to methodology with the aim of learning 
different approaches in analyzing hashtag feminist activism that is practiced in online platforms. 
The inclusion criteria of the studies were as follows: a) studies that utilized feminist media 
research; b) the articles that were published in 2017-2019; c) studies that explicitly highlighted 
teaching and learning aspects of the hashtag activism. The articles that did not explicitly talk 
about teaching and learning/raising consciousness awareness aspects of digital feminism were 
excluded. 

Since the search engine ERIC (EBSCO) did not sufficient number of relevant resources 
for the review, Google Scholar was used, targeting peer-reviewed journal articles and excluding 
dissertations and books that are not available open source or through the NIU library database. 
Over 40 entries were scanned for eligibility before the ten articles were selected. The following 
research questions were formulated to analyze the content of the articles: 

1. Within research that highlighted teaching and learning aspects, what is the context of 
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their foci? 
2. Which theoretical framework(s) di the studies use? 
3. What methodologies did the studies employ? 


Findings 


To analyze the content of the seminal studies, a free and open source software- Docear 
was used. Docear is a new solution to academic literature management allowing the most 
comprehensive organization of literature. As this software offers PDF metadata retrieval, it 
allowed me to transfer the annotations from PDF files selected for the study. This process helped 
me identify the similarity of words/concepts related to teaching and learning aspects featured in 
the study. 

Summary of Findings for RQ1 


In relation to RQ], teaching and learning as well as consciousness raising as the 
characteristic of research has been identified as the foci of the selected studies. The content of 
the articles highlighted educational aspects of the blogs through which feminist movements, 
using the feminist mantra “personal is political,’ empowered women to share their experiences 
with racism, sexism and other “isms” and to create collective identities in their fight against 
social injustice in relation to women. Some of the research highlighted the connection between 
the second and firth wave feminism, stating that “personal is political” slogan of the second wave 
feminism was used in the discussions online feminist movements. The phrase was coined by 
Carole Hanisch in 1969 and later was recuperated by Audrey Lorde (Troutman, 2014). 

Personal is political. To describe how personal stories shared in digital platforms have 
gained political meanings, the majority of the articles discussed different contexts of application 
of the popular feminist slogan “personal is political” in the fourth-wave feminism. In those 
digital spaces, women shared individual stories related to sexual assault, rape and other issues to 
raise consciousness among the online community and feminist circles, thus proving the 
appropriateness of the slogan in digital platforms. For instance, Aitken’s (2017) article opened 
up a conversation about the empowering and consciousness raising power of the fourth-wave 
feminism through relevant examples. As she noted, the use of the second wave feminists’ famous 
slogan “personal is political” in the fourth-wave digital movements connected this wave to the 
second wave of feminism because shared personal stories raised salient political issues. Although 
Aitken (2017) talked about active digital movements in general without focusing on the specific 
hashtag movement, she underscored the power of digital tools in amplifying the diverse voices in 
inclusive global platforms. Online platforms have the power of unite and inform women, 
especially young women, discovering feminism and developing collective identities across 
digital spaces on issues that resonate with their experience (Aitken, 2017). Drawing a parallel 
between second wave and the fourth wave feminism, Zimmerman (2017) explained that the 
important message of the second wave, which states “personal is political,” as a tool for 
collective consciousness raising, was transferred to the fourth wave in that digital technology has 
become organizing and consciousness raising tool. To highlight the importance of the Twitter 
community in the analysis of the discourses on racism, feminism, and online representation, 
Zimmerman (2017) traced the genealogy of the fourth-wave feminism, noting that the this 
movement combines the important elements of the second and third wave feminism, such as 
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resistance to oppression of the second wave and the third wave’s recognition of digital culture to 
facilitate feminist discourses. Initially taking a critical stance on the political role of the fourth- 
wave feminism, Rogan and Budgeon (2018) posed a question about the applicability of the 
second-wave movements definitive ‘personal is political’ slogan to digital feminism. Collecting 
data through the questionnaire posted on Facebook and Twitter from young women under the 
age of 21 across the UK, posted on social media, the authors drew conclusion that about the 
importance of social media providing a political space that is consistent with the definitions of 
the political advocated by the second wave feminist, while helping women define their personal 
identities through the connection with others who shared similar experiences with them through 
Facebook events and tweets engage with politics. Discussing experiences of young girls at a 
secondary school in England with digital feminism, Kim and Ringrose (2018) recommended that 
the social media could serve as a strategic tool for exploring rape culture and other topics. The 
girls joined feminist movements for personal reasons, such as raising their consciousness about 
feminism. Through their engagement with online feminism, they could challenge “notions of 
youth as politically indifferent and inept” (p. 47). 

Focusing on the feminist analysis of hashtag movements, several other articles 
highlighted the links between trending (personal) topics with a larger structural framework 
(political). For example, analyzing frequently used terms in #YesAllWomen, Barker-Plummer 
and Barker-Plummer (2017) notes that this hashtag feminism is both personal and political 
discourse because the majority of the shared stories comprises first-person accounts on 
oppression, gendered violence and sexual assault. The hashtag #YesAllWomen reinforces a 
feminist rhetoric links trending topics with a larger structural framework, 1.e., creates links 
between the personal and the political. Through the analysis of discourses of hashtags #MeToo 
and #BeenRapedNeverReported movements, Mendes, Ringrose and Keller (2017) discuss how 
these popular online feminist movements create safe spaces that challenge sexism, patriarchy and 
other forms of oppression via digital platforms. Extending this conversation to #MosqueMeToo 
movement, Point’s (2019) article examines the potential of this hashtag feminism for 
empowering Muslim as it encompasses the real-life experiences and views of Muslim women 
across the globe. The unique aspect of this movement is that it analyzes feminist and religious 
discourses together, raising consciousness about the interrelatedness of these topics in regard to 
Muslim women’s experiences. Turley and Fisher (2018) discuss British feminists’ hashtags, 
namely #everydaysexism and #shoutingback, for their individual protests against unequal 
treatment of women in the patriarchal culture while advocating for issues that affect women 
around the world. Highlighting the importance of social networking sites for fostering feminist 
movements, the authors note that digital spaces are vital for practicing the fourth wave of 
feminism to creating global communities. Taking a different stance on digital feminism, Jester 
(2018) analyzes the three curriculum groups (Rhodes Must Fall, Why is My Curriculum White?, 
and Women Also Know Stuff) to underscore the importance of representation of diverse 
experiences in the dominant Western academic discourse, which operate to bring changes to 
higher education. Through Twitter and Facebook, these movements have become successful in 
bringing to the forefront the curriculum issues to the attention of higher education practitioners, 
academicians and policy makers. Therefore, Jester (2018) proposes that scholarship on the 
feminist fourth wave can be enhanced through the examination of these groups as some 
successful events offer promising opportunities. For example, re-tweets and hashtags allowed 
showcasing the work of women in history (@womnknowhistory (through Women Also Know 
Stuff website’s ‘Women Also Tweet’ section) while global Facebook groups of Why is My 
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Curriculum White? helped organize events garner support and garnering support for making 
curricula diverse and representative of all groups of people. Ringrose (2017) also discusses the 
benefits of using Twitter as a teaching tool in higher education contexts, detailing her a lecture 
where she deconstructed her personal activism through Twitter account. Her participation in 
major feminist Twitter discourses about the U.S. election in 2016 helped her to initiate a debate 
on raising awareness among her students about the importance of fostering the feminist slogan 
the ‘the personal is political’ through tweet cultures. 


Summary of Findings of RQ2 


In relation to the findings of RQ2, the findings revealed two major themes: (a) 
intersectionality as a framework; (b) feminist pedagogy and (c) Freire’s Pedagogy as a 
theoretical basis. 

Intersectionality as a framework. Although Aitken (2017) does not frame her study 
within the intersectionality, she recommends that combination of community education with 
feminist pedagogies can help raise consciousness about intersectional identities. Criticizing 
#YesAllWomen from the intersectional feminist lens, Barker-Plummer and Barker-Plummer 
(2017) point out to the movement’s failure to be adequately intersectional due to its dominant 
white discourse thereby making intersectional feminists set up parallel hashtags. Jester (2018) 
argues that intersectionality is the foundational principle of curriculum activism is a core 
component of fourth-wave feminism and within the higher education context, it is. Drawing from 
the works of Tara L. Conley and Kimberlé Crenshaw, Point (2018) explores the potential of 
hashtag movements to raise intersectional awareness about Muslim women’s struggles for 
equality, contributing to the debates regarding gender and Islam, and transnational feminist 
studies. Although Turley and Fisher (2018) do not use the intersectionality framework, they 
underscored the importance of social media campaigns for fighting against sexism, misogyny 
and rape culture through discussions. To enhance the discussion, they recommend analyzing 
online feminism through the lens of transdisciplinary literature, namely national, transnational 
and global academic literature. Drawing on Black feminist and intersectional feminist 
scholarship, Ringrose (2017) explores racism and misogyny in the debates on Twitter 
surrounding the 2016 election in the United States. Mendes et al. (2017) also recommend 
reviewing public discourse on Twitter with intersectional lens because this framework allows 
understanding the accessibility to the discourse in terms of ability, race and class privilege, 
which is the main aspect of digital feminism. Zimmerman (2017) also draws on intersectional 
scholars’ works, noting that intersectionality serves as the political driver and theoretical 
framework employed by fourth wave feminists. According to Zimmerman, this framework is 
necessary for analyzing hashtag movements to foster dialogues on intersectional identities and 
their representation in online platforms. 

Feminist pedagogy. All the articles reviewed highlighted explicitly or implicitly the 
importance of feminist pedagogy to understand the discourse happening in hashtag movements. 
Therefore, it is important how the scholars describe feminist pedagogy in those articles. For 
instance, Aitken (2017) summarizes research on feminist pedagogies by drawing attention to 
relationship between community education and feminist pedagogical practice. Critically 
analyzing feminist teaching and consciousness raising practices, the author explored their 
consistency with a community education approach. Focusing on the fourth-wave feminism, 
Aitken (2017) underscores the power of online platforms to evoke debates on gender inequality 
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through campaigns and to amplify feminist voices. Trying to explain how feminist pedagogy is 
practiced through a hashtag movement, Barker-Plummer and Barker-Plummer (2017) draws 
attention the discursive activism that took on the platform in two forms: (a) a site for collective 
identity to raise consciousness (sharing of experiences and stories) and (b) recirculation of shared 
stories and experiences through other platforms, affecting public discourse beyond Twitter. 
Although not explicitly, Point’s (2018) study notes that the fourth-wave feminism focuses 
affords a contemporary method of community building, namely through the sharing of collective 
experience. When discussing the importance of feminist pedagogy in raising consciousness about 
racism and sexism through Twitter, Ringrose (2017) reflects on the dynamics of her own 
participation in Twitter as a public medium for practicing the feminist pedagogy. 

Pedagogy of Oppressed as a theoretical basis of feminist pedagogy. Some articles 
trace feminist pedagogy back to Freire’s Pedagogy of the Oppressed and comment on the 
importance of the pedagogy in exploring feminist issues discussed in their studies. Indeed, 
Pedagogy of the Oppressed provides an appropriate departure point for feminist pedagogies 
because the knowledge production is a dynamic dialogue in which the learner is directly 
involved (Freire, 2005). Citing some scholars works, for instance, Aitken (2017) notes that 
feminist pedagogies replenished Freire’s work, putting an emphasis on gender in traditional 
educational contexts and creating inclusive framework that aims to restructure education as a 
space for all genders. Jester (2018) argues saying that Freire’s Pedagogy of the Oppressed allows 
us to understand the feminist components of curriculum activist groups as it forms the basis of 
the groups, specifically of Why is My Curriculum White? 


Summary of Findings of RQ3 


In relation to RQ3, I have based classification on the authors’ descriptions of their 
methodology, rather than applying a priori categories. The content analysis of blogs and groups 
revealed that the primary purpose of them were to teach, educate and raise consciousness of 
women about the existing problems. For example, Barker-Plummer and Barker-Plummer’s 
(2017) case study’s methodology was the content and discourse analysis of #YesAllWomen 
hashtag, including two million tweets and of 251 media and blog stories, the interactions of 
which are traced back to the hashtag. The rationale for choosing #YesAllWomen as a case study 
explained with the hashtag’s potential for underscoring the importance of online discursive 
politics for public debate ad digital protest. Second, #YesAllWomen is a successful example of 
how feminists are developing new strategies to engage and utilize emergent communications 
platforms. 

Other studies used different methodologies to explore teaching aspects of the social 
media discourses or reviewed the literature that discussed the potential of online platforms in 
uniting women across the globe. The table below provides detailed information about the 
methodology of the studies and outcomes reported in those studies, highlighting the importance 
of integrating non-academic tools to higher education to foster the inclusion of intersectional 
identities in the mainstream academic discourse. 
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Table 1. Review of the Methodology of Articles in SLR 


Articles Methods Participants Results reported 
regarding knowledge 
production/raising 
consciousness 

Aitken, M. (2017). Feminism: A Areviewof N/A “The fourth wave 


fourth to be reckoned with? 
Reviving community education 
feminist pedagogies in a digital 
age. Concept, 8(1), 1-18. 


Barker-Plummer, B. & Barker- 
Plummer, D. (2017). Twitter as a 
feminist resource: 

#Y esAllWomen, digital 
platforms, and discursive social 
change. In J. Earl and D.A. 
Rohlinger (Eds.), Social 
movements and media. (pp. 91- 
118). Bingley, UK: Emerald 
Publishing Limited. 


Jester, N. (2018). Representation 
within higher education 
curricula: contextualising and 
advocating for feminist digital 
activism. Teaching in Higher 
Education, 23(5), 606-618. 


research that 
recommends 
feminist 
pedagogical 
intervention 
as a possible 
method to 
combine the 
ideas of the 
fourth wave 
feminism and 
community 
education. 


Content and 
discourse 
analysis 
methods to 
analyze the 
#YesAllWom 
en hashtag 
qualitative 
coding of the 
news and 
blog stories 


Content 
analysis of 
Rhodes Must 
Fall, Why is 
My 
Curriculum 
White? and 
Women Also 
Know Stuff 
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An analysis 
of a sample 
of 251 media 
and blog 


Actual 
number of 
participants 
contributed 
to the 
movements is 
not reported 


deconstructs individualist 
discourses and, in true 
feminist spirit, makes the 
personal political (p.7). 
Community educators have 
a duty to resist agendas that 
seek to silence women’s 
voices and through their 
practice for social change. 


Participants in discourse 
focused on core feminist 
concerns — rape, 
harassment, sexual 
entitlement and inequality 
All key elements in 
discursive activism were 
revealed, such as building 
collective 
identity/collective 
consciousness. 


Curriculum movements 
offer promising 
opportunities for higher 
education. Followers of the 
movements reported that 
higher education 
institutions are starting to 
change their curricula 


Kim, C., & Ringrose, J. (2018). 
“Stumbling Upon Feminism”: 
Teenage girls’ forays into digital 
and school-based 

feminisms. Girlhood 

Studies, 11(2), 46-62. 


Mendes, K., Ringrose, J., & 
Keller, J. (2018). # MeToo and 
the promise and pitfalls of 
challenging rape culture through 
digital feminist 

activism. European Journal of 


Women's Studies, 25(2), 236-246. 


Point, C. (2019). # 
MosqueMeToo: Islamic 
Feminism in the Twitter sphere. 


Ringrose, J. (2018). Digital 
feminist pedagogy and post-truth 
misogyny. Teaching in Higher 
Education, 23(5), 647-656. 


Research on 
participants’ 
activism on 
social media 
platforms 
(Facebook, 
Instagram, 
Tumblr, and 
Twitter) to 
explore their 
postings over 
four-week 
period 


Over 800 
pieces of 
digital 
content, 
including 
blog posts, 
tweets, and 
selfies. 


#MosqueMe 
Too as an 
artifact to 
analyze 
Twitter 
dsicourse 
The 
experience of 
having the 
feminist 
content of 
personal 
public 
Twitter 
account 
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Qualitative 
study: 
Interviews 
with four 
most active 
members of 
the feminist 
society and 
artifact 
analysis of 
their 
accounts in 
social media 


Feminist 
media 
research and 
qualitative 
study (the 
views of 82 
girls, women 
and men who 
contributed 
to the 
blogs/tweets) 


No actual 
number of 
screenshots 
from Twitter 
presented 


The author 
used the 
digital 
affordances 
of hashtags 
and trends on 
Twitter to 
spread a 
message, as it 
offers a 
unique 
pedagogical 
dimension 
for feminists 


The failure of the school to 
use non-academic 
pedagogies via social media 
engagements as it is 
important—even 
transformative—sources of 
learning 


Despite fear of attack and 
experiences of trolling, 
digital feminism can be 
experienced as extremely 
positive in generating 
community, connection, 
and support for feminist 
views, and solidarity in 
calling out rape culture 


#MosqueMeToo serves as a 
statement of justice, 
requiring a demand for both 
intervention and visibility 
of Muslim women. 


Anti-feminist threats travel 
beyond the digital sphere 
into the personal lives as 
feminist academics 


Rogan, F., & Budgeon, S. (2018). 
The personal is political: 
Assessing feminist fundamentals 
in the digital age. Social 
Sciences, 7(8), 1-19. 


Turley, E., & Fisher, J. (2018). 
Tweeting back while shouting 
back: Social media and feminist 
activism. Feminism & 
Psychology, 28(1), 128-132. 


Zimmerman, T. (2017). # 
Intersectionality: The fourth 
wave feminist twitter 
Community. Atlantis: Critical 
Studies in Gender, Culture & 
Social Justice, 38(1), 54-70. 


Mixed 
methods 
study focused 
on a set of 
important 
questions 
regarding the 
nature of 
young 
women’s 
relationship 
to feminism 
in the age of 
digital 
femininities 


Review of 
two social 
media-based 
feminist 
campaigns 


Review of 
scholarship 
on genealogy 
of the fourth 
wave 
feminism and 
the analysis 
of the term 
“intersectiona 
lity” 
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to raise 
awareness 
and 
spreading 
ideas around 
gender and 
sexual 
violence 


50 responses 
from young 
women under 
the age of 21. 
The nine 
focus groups 
in three urban 
locations in 
England— 
one in the 
South-East, 
one in the 
West 
Midlands and 
one in the 
North West 


N/A 


N/A 


The ‘personal is political’ 
falls under four interrelated 
categories: power, the 
private/public dichotomy, 
political action, and 
subjectivity. Each of these 
key dimensions remains 
central to a contemporary 
feminist analysis of the 
experiences of young 
women’s engagement with 
digital practices. 


Social media is not the only 
mode of online feminism; 
importance and value of 
offline feminist movements 
should not be ignored. 


Intersectionality is the most 
appropriate theoretical 
framework for the fourth 
wave movement for its 
potential to practice 
political intervention and 
visibility without 
segregation and/or 
marginalization of people 


As it is seen from the table above, some authors of the articles were analyzing the content 
of the blogs/groups, being engaged in feminist media research. Contemporary feminist 
researchers are engaged in feminist media research to better understand how power dynamics is 
represented in media. In this process, the researchers get connected to the text, and “become [s] 
part of the power structure that informs the text” (Hesse-Biber, 2014, p. 287). Getting connected 
to the text offers some advantages and disadvantages. The advantages can be explained by 
discovering multiple meanings of the text and having less interaction with people, and the 
disadvantages are mainly about how texts are perceived by the audience. This is a laborious 
process for researchers to analyze as they can only make assumptions. However, feminist media 
research has a certain role in social transformation as it aims at empowering women through 
feminist discourse. 

Conclusion 


The findings of the study underscore the importance of developing pedagogy centered on 
anonymous social media spaces. The online aggressive behavior should not be tolerated as it 
tries to suppress the democratic discourses. Educational practitioners, university professors and 
rhetoric scholars are, therefore, charged with an important task, and that is to participate in 
anonymous and pseudonymous digital social media to reveal opportunities for democratic 
participation using the discourse of digital media to foster active feminist movements. 
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Abstract 


This qualitative study tends to explore what accessibility design can be most important to 
facilitate learning in an online course for postsecondary students with low vision. The study is 
conducted in a U.S. public university offering online courses in the Midwest. This study was 
guided by two research questions: (1) What accessibility design did students with low vision 
who experienced online courses perceive to be helpful for their learning? (2) What accessible 
features would students with low vision want to exist in future online courses? The theoretical 
framework for this study was Universal Design for Learning (UDL). Three participants were 
interviewed to share their experience with online learning and to explore which accessibility 
aspects were perceived the most helpful for students with low vision. The finding revealed that 
alternative formats for materials—such as Word documents or Rich Text formats (RTF) and 
accessible PDF files—were the most helpful accessible text format in the online courses. In 
addition, the finding showed headings and color contrasting for the online content are the main 
aspects of design to increase accessibility and to facilitate reading for students with low vision. 
The last finding revealed that students with low vision need two additional accessibility design to 
be employed in online courses: audio response and instructor video. 


Keywords — Online learning, Accessibility, Low-Vision, Visual impaired 


1. Introduction 


In the last two decades, due to flexibility and accessibility, online courses are becoming 
increasingly popular among non-traditional students and learners who have (in)visible disabilities 
(Summers et al., 2014). Students with disabilities need support, particularly, students with low 
vision need special accessible features in online courses because they have some difficulties 
dealing with the technologies (Crow, 2008; Fichten et al., 2009; Summers et al.,2014). Universal 
Design of Learning (UDL) provides several accommodations for postsecondary students with low 
vision through using assistive technology and providing a variety of accessible features for this 
type of vulnerable population (Crow, 2008). Relevant research (Lorenzin & Wittich 2019; Okiki, 
2019) shows that low vision students will succeed academically when they take online courses 
with proper accessibility design. To explore which types of accessibility design aspects, based on 
the UDL principles, are deemed to be appropriate for online settings, this qualitative study seeks 
to understand the perceptions of students with low vision. 


2. Literature Review 
2.1.Online Courses 
Online courses attempt to create a type of learning environment and serve as a process of 
connecting students, instructors, and learning resources when they are not physically present in the 
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same location (Park & Choi, 2009). In 1997, the first online course platform was launched at 
famous universities, such as Yale, Cornel, and University of Pittsburgh. In the same year, a 
Learning Management System (LMS) known as Blackboard™ was founded and has become 
widespread to deliver online instruction and it is still utilized in many educational institutions and 
universities across the globe (Morton, 2016). Online courses use asynchronous and synchronous 
technologies. Synchronous technology requires students and instructors to work at the same time 
but not in the same place through using video conference (Palmer, 2012). In contrast, asynchronous 
technology does not require students and instructors to work at the same time (Palmer, 2012). 
They can work independently at a convenient scheduled time for each of them. 

In the last three decades, online courses have significantly increased in higher education 
(Betts et al., 2013). Recently, 30% of postsecondary students are enrolled in at least one online 
course in one of the U.S higher education institutions (Cole et al., 2014). Although online courses 
have increased, students with disabilities enrolling in institutions of higher education have also 
increased over the last twenty-five years (Lyman et al., 2016). Higher education has attempted to 
make online courses more effective and accessible for all students, however, some instructors 
and/or institutions may overlook the needs of students with disabilities (Kharade & Peese, 2012). 
Cook and Gladhart (2002) stated that 10% to 15% of postsecondary students identify themselves 
as disabled. According to the American Disabilities Act (ADA), a disability is a physical or mental 
impairment that substantially limits one or more major life activities. To be labeled as disabled, a 
person must have a history or record of such an impairment, or a person should be perceived by 
others as having such an impairment. These self-identified students with disabilities should have 
equal opportunities in their online courses as other students. 


2.2.Online Courses for Students with Low Vision 

Low vision is one of the common types of visual disabilities (Richardson, 2014). It is 
defined as the functional limitation of the eye or eyes or the vision system (The American 
Foundation for the Blind, 2015). The American Foundation for the Blind (AFB) defines low vision 
as a condition caused by eye disease in which visual acuity is 20/70 or poorer in the better-seeing 
eye and cannot be corrected or improved with regular eyeglasses (AFB, 2015). Students with low 
vision usually have several academic difficulties (Moola, 2015). One of these difficulties is using 
technology because sometimes they cannot adjust technology according to their needs. For purpose 
of this study, low vision identifies as “a person who has difficulty accomplishing visual tasks, even 
with prescribed corrective lenses, but who can enhance his or her ability to accomplish these tasks 
with the use of compensatory visual strategies, low vision and other devices, and environmental 
modifications” (Corn & Koenig, 1996, p.4). 

Consequently, the emergence of online courses has brought challenges for students with 
low vision (Argyropoulos et al., 2019; Summers et al., 2014). The literature on the experiences of 
students with low vision is scarce, and most seminal articles focus on students with disabilities 
without specifying the type of disability (Lorenzin & Wittich 2019; Okiki, 2019). However, some 
relevant studies (e.g., Lee & Oh, 2017; Richardson, 2014) had observed that students with low 
vision are not often active in online courses due to the challenges they face in interacting with 
learning materials. On the contrary, several studies suggest online courses are beneficial for 
students with low vision since they provide remote learning experience (Barnard et al.,2012; 
Haegele et al., 2018; Kharade & Peese, 2012) and allow instructors to provide remote instructional 
assistance to the students anytime and anywhere even if they live far from the main campuses of 
the universities (Holmgren, 2018). 
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Online courses benefit students with low vision because they can find a solution for the 
challenges attending physically on campus, which poses great difficulty for them (Kharade & 
Peesa, 2012; William et al., 2006). Kharade and Peesa (2012) addressed that the flexibility in the 
location, scheduling, and delivery of online courses reduced the challenges for attending on 
campus by providing flexibility in time and place of delivery. Feucht and Holmgren (2018) 
reported that students with low vision drop out because they cannot drive to the campus and do 
not live close to the campus. Walking around campus is also a challenge because sometimes it 
requires students with a very low vision to use aids such as a cane or a guide dog. This is because 
in some cases students with low vision cannot even see the small things, or in other cases, students 
with low vision cannot see things in bright or dark places. Therefore, they often have a difficult 
time self-navigating outside of well-known environments and prefer to be indoors (Long et al., 
1990), and some prefer to study and work in small physical spaces (Haegele et al., 2018). Asa 
result, low vision affects a person’s ability to learn or perform many job duties, which severely 
limits his/her main life opportunities for education and employment (Long et al., 1990). Therefore, 
online courses became such a great option for students with low vision to complete their 
educational degrees and be more motivated to succeed (Kharade & Peese, 2012). 

Besides flexibility, online courses allow students with low vision to adjust the instructional 
material through assistive technologies according to their needs (Crow, 2008; Fichten et al., 2009) 
during learning, reading, writing and acquiring academic and nonacademic skills (Hewett et al., 
2017; Rosner & Perlman, 2018). In addition, using assistive technologies in online courses help 
students facilitate learning and receive equal learning opportunities (Hewett et al., 2017). Because 
of this equality, students with low vision can be more active and motivated to participate in online 
activities such as discussion and group work. Assistive technologies help to improve the quality 
of learning for students with low vision (Crow, 2008). Online courses with assistive technologies 
encourage students with low vision to be active participants and share the ideas with classmates 
and instructors remotely in online course activities (Crow, 2008; Fichten et al., 2009; Hewett et 
al., 2017). 


2.3.Accessibility 

Accessibility is an important priority in online courses delivered by top universities such 
as Harvard University, UC Berkeley, and MIT (Alahmadi, 2017). Following their trend, many 
colleges and universities have started to make program and policy changes in their online courses 
(Zuriff, 1996). Accessibility addresses the design of technology rather than the needs of specific 
individuals (Alahmadi, 2017). Accessibility means using course materials and tools by all types of 
students, regardless of their physical and/or developmental impairments. When a course is 
accessible, most of the students even those with disabilities can reach the material equally. All can 
get access to the course delivery system, navigate the course content, submit assignments, and 
successfully use all course tools. The most common example of accessibility includes obtaining 
printed materials in alternate formats (Pittman, et al., 2014). Other examples involve the inclusion 
of a statement of support for students with disabilities in the course syllabus. In addition, all video 
content (web, DVD, and VHS) should be captioned, and transcripts of audio-based material and 
video-based materials should be available if they cannot be captioned. 

There are several benefits of accessibility in online courses. The accessibility allows 
students to use flexible materials that can be adjusted according to their special needs and 
preferences (McKenna & Velasco, 2018; Pittman, et al., 2014). Audio, images, graphics, 
animations, video, or text which are often the tools to present information and the relationships 
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between objects, actions, numbers, or events. However, visual representations are not equally 
accessible to all students, particularly visually impaired students (McKenna & Velasco, 2018). 


3. Theoretical Framework 

Most institutions of higher education in the U.S. incorporate the principles of Universal 
Design for Learning (UDL) into the educational and instructional materials. UDL is a framework 
for improving instruction because it helps provide equal opportunities for all learners to succeed. 
This strategy provides flexibility in how learners to access, engage with and demonstrate what they 
understand and increases the quality of learning materials for everyone (Rose & Mayer, 2008). 
UDL principles support students with low vision who have some challenges in online courses by 
providing resource and flexibility access to engage the students complete learning (Houston, 
2018). Most of the research has found that UDL is essential for integrating students with visual 
impairments into higher education (Al-Azawei et al., 2016; Houston, 2018; McKenna & Velasco, 
2018). According to CAST (2008) and Rose and Mayer (2008), there are three UDL principles: 
representation, action and expression and engagement. The first principle of UDL is 
“Representation, ” which involves providing learners with various ways of acquiring information 
and knowledge that have a connection to the accessibility formats, which require instructors to 
provide various resources to facilitate students’ access to the learning materials. The second 
principle is “Action and Expression,” which provides students with various routes to access the 
necessary materials using assistive technology. The third principle is “Engagement,” which 
enables an instructor to tap into students’ interests, challenges them appropriately, and motivates 
them to learn through facilitating resources. 

This study sought to explore the perceptions of the current experiences of students with 
low vision in online courses to identify what accessibility design aspects offer the greatest support 
based on UDL guidelines and would be beneficial. This research is intended to provide 
recommendations for future instructors and instructional designers to consider when creating 
online courses for students with low vision. 


4. Methodology 
4.1.Setting, Sample and Participants 

The study took place at a U.S. public university in the Midwest with a total enrollment of 
students 17,169 for Fall 2018. According to the university website (2018), there are 12,788 
undergraduates, 4,121 graduates and 260 college of law. As the mission states, the school 
celebrates its diverse population in all its forms, including gender, race, ethnicity, ability, 
Spirituality, sexuality, age, and individual identities. This Midwest public university offers 
approximately 20 undergraduate and graduate degrees online and about 10 additional certificates 
fully online. 

For this study, purposeful sampling was used because the researcher purposefully selected 
the students with low vision who were enrolled in online course settings. Creswell (2009) stated 
that in qualitative research, researchers identify participants and sites using purposeful sampling 
based on places and people that can best help a researcher understand the central phenomenon. 
The participants were from different major and educational level and ages range from 22 to 54 
years old. All the participants registered in the disability resources center (DRC) and had number 
of online courses taken between 3 to 6 courses. The participants had different low vision types; 
Retinitis Pigmentosa, Optic Nerve Coloboma, and Blurred Vision. 
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4.2.Research Questions 
1) What accessibility design did students with low vision who experienced online courses 
perceive to be helpful for their learning? 
2) What accessible features would students with low vision want to exist in future online 
courses? 


4.3.Data Collection 
To obtain data, three students were interviewed, and the interviews were audio-recorded. 
Interviews were conducted either face-to-face or by telephone depending on the participants’ 
preference. The duration of each interview was 45 to 60 minutes. They were asked around 20 
questions, consisting demographic questions, questions related to their experience with online 
courses, and questions regarding accessibility and assistive technologies that helped them 
overcome their challenges. 


4.4.Data Analysis 

Using a professional transcription service (Rev.com) to transcribe the interviews. Then, the 
textual data of the interviews has been read multiple times to gain a deeper understanding on 
information contributed by participants (Creswell, 2012). Prior to proceeding to the data analysis, 
a codebook was created based on each research question: accessibility coded as AA and 
highlighted in pink; assistive technology coded as AT and highlighted in blue; and the wish list of 
the students with low vision coded as WLA and highlighted in red. The data had been analyzed 
line-by-line to code thoroughly. Then, applying an open-coding strategy to analyze the responses 
from the interviews by looking specifically for words that are related to pre-defined codes. For 
example, when the participants mentioned a screen reader, the researcher coded it as AT and 
highlighted it in blue. According to Patton (2002), the process of coding starts with segmenting 
and labeling similar codes to form descriptions and broad themes. Therefore, each of the pre- 
defined codes labeled as themes. The themes were used to respond to each research question. 


5. Findings 
5.1.What accessibility design did students with low vision who experienced 
online courses perceive to be helpful for their learning? 


The participants identified three accessibility design aspects: alternative formats for 
materials, headings, and color contrasting for online content as the most helpful accessibility 
regarding their learning experiences. Ruby responded that she has a lot of reading requirements 
to complete the online assignments as a graduate student. She clarified, “Word document and RTF 
are the most beneficial types of alternative formats for textual online materials. Having formats 
like word documents or RTF are super helpful to access the text and use Read Aloud feature when 
I need it.” 

Sarah added, “I avoided reading. I avoided doing anything that wasn't just hands-on like 
the building because I'm actually in the construction trades.” Also, she mentioned that she has 
difficulty reading a book but she can read text font “Arial’ and size 16 or 18. She said, “sometimes 
I get notes, teacher's notes and IJ actually have to change the font to just an Arial font because the 
New Times Roman is hard for me to read. Yeah, it has too many like little curves in it and the 


27 


letters are too close.” Thus, she requests alternative formats for the online textual materials to 
audio: “Well, everything needs to be audio for me.” 

In contrast, Karen described that she prefers Adobe accessible PDF version, which includes 
features that allow students with low vision access text to be more readable such as taking notes, 
searchable text, and tracking of information. 


I would like to see that used more and more like accessible PDF documents instead of just 
like taking a picture. I think it's just giving me more access. For example, being able to 
look at like a PowerPoint in an accessible format. I can use it more easily to take notes 
and to keep track of information instead of having to like struggle through the slide. 
Sarah and Ruby use screen reader software, which they mentioned as the most helpful assistive 
technology for them. Ruby said: 


And so being able to have, like for example, the articles that we had to read were in two 
formats. They were in like a scanned in PDF and they were in like a word document. And so I 
was able to use my screen reader to read the word document and I had access to the course 
material without having to ask somebody to help me read it or help me scan it and to be able 
to change the scanned document. 


Moreover, Sarah and Ruby use the “Read Aloud” feature in Word. For the web pages and other 
documents, they use screen reader software “Narrator” in Windows. They mentioned that they do 
not install JWAS or Kurzweil 3000 on their computers. Ruby has used JAWS in the past; however, 
she is not using JAWS anymore because the screen reader is available in Windows and helps her 
read long articles. 

Additionally, the participants identified heading and color contrasting for online courses 
that helped them find and use online course materials. Ruby and Karen mentioned that designing 
online courses with headings guide them during navigating the online environment. Ruby said, 
“T'd say headings help split things into sections.” Karen said, “have a lot of headings to navigate 
that makes things easier.” In addition, Sarah mentioned that using contrast color for the text and 
background facilitates reading the PowerPoints. She described her current challenge reading some 
colors: 


One of the classes I have now, he has a lecture, it's short, it's a simple lecture. But then he 
has a PowerPoint and that does not have any audio to it and it's kind of a struggle for me 
to read through that. It's actually on the university [brand]; the background is red with the 
black and the gray and that's actually hard to read. I think that's hard to read. 


An additional accessible design that one of the interview participants reported was added 
description for videos. Sarah said, “I'd say more ideal description from videos” would help students 
with low vision understand what is happening in the video. 


5.2.What accessible features would students with low vision want to exist in future 
online courses? 


The results revealed some of the improvement accessibility suggestions that students with 
low vision wished would exist in online courses. Audio and instructor video were the two 
alternative formats that did not exist in most of online course. The participants described two ways 
of using audio in online courses: audio with PowerPoint and audio response in the discussion 
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boards. Sarah suggested that instructors in online courses should use audio with PowerPoint slides 
to facilitate learning; she said “when the teacher has a PowerPoint, it's great, but I have to read it. 
I want it to read to me and I want the word to stand out as are being read.” Ruby suggested adding 
the option of “audio” to participate in the discussion boards; she said: 


I think more audio would be really helpful. So I don't know if this is something that 
necessarily instructors would have jurisdiction over, but I guess just having it's different 
alternatives, communicating with discussion boards, you know maybe having like an audio 
option to leave audio responses. They're having a more simplified platform. So that would 
be one of the things on my wish list. 


Additionally, all the participants suggested that instructors in online courses should record 
videos to help the students be engaged in their learning. Karen would have more videos to 
understand some subjects; however, she did not specify the video types. She said, “Say we were 
assigned to read a chapter and then the teacher would have provided a video or something 
explaining certain things. I think that's always helpful.” On the other hand, Ruby and Sarah 
specified to receive instructor-recoded videos. Ruby said, “I wish the professors would do in- 
person videotaping of themselves.” Moreover, Sarah added that seeing the body language helps to 
engage the students in online courses; she said, 


When the instructor goes into the connect and has a PowerPoint and he's just talking, I 
think that's okay. But it would be really nice if you actually saw him because movement, 
your body gestures are engaging. 


Sarah also recommended the instructors should record video to explain the course content to 

improve the students’ learning performance; she said, 
Everything was online. It would have been so cool if the teacher had done what my 
classroom teacher did and said, “Okay, here's 20 minutes, here's the problem on the 
chalkboard. I'm video recording myself and this is what you do and, oh, you think about 
this and now, you go to the next step and you have to remember that. And then ...”’ That 
would have been great. I mean I know that some minor technical classes, safety will say, 
“Look, my last class was a safety class.”’ I mean, I can't imagine an instructor going, if I 
taught the class ... Ultimately I would like to teach but if I taught the class, I probably 
would read the book, they're like safety hazards. So let's say in the parking lot or in a 
building, I probably would record it. “This is a safety hazard.” 


6. Discussion 
6.1. Accessibility 


There were three major findings related to accessibly. The first finding revealed that 
alternative formats for materials—such as Word documents or Rich Text formats (RTF) and 
Adobe accessible PDF files—were the most helpful accessible format in the online courses. These 
alternative formats allowed students with low vision to make changes according to their own needs 
and use text-to-speech assistive technology such as screen reader software or the “Read Aloud” 
Word file feature. All participants highlighted that the alternative formats provided them equal 
access to the online materials. In addition, this finding is consistent with literature (e.g. Pascual, 
2014; Spooner, 2014) mentioned students with low vision preferred to use alternative formats 
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because it allows them to edit and make changes that best suits their needs. For example, Sarah 
explained that she could only read the “Arial” font; therefore, having the materials in Word allowed 
her to change the font to “Arial” because “Times New Roman” was hard for her to read. This 
finding is consistent with Houston’s (2018) study that recommends using Sans-Serif fonts in online 
course materials because Serif font types help make online content more legible to all students, 
including those with visual impairments. Common Sans-Serif fonts include Arial, Trebuchet, and 
Helvetica. On the other hand, some Serif font types—such as Times New Roman, Courier, New 
Century Schoolbook, and Palatino—have semi-structural details or small decorative curves on the 
ends of some of the strokes, making the letters and symbols challenging to read. Overall, this 
finding aligns with UDL “representation” principles, which entail the accessibility of instructional 
materials for all students, including students with disabilities, providing them equal access (CAST, 
2008). 


The second finding was that participants identified assistive technology within alternative 
formats as one of the most helpful accessibility features. The finding showed that text-to-speech 
reader software, such as screen reader, was the most helpful assistive technology for students with 
low vision to read online course materials. For instance, Sarah mentioned that she always needed 
to use assistive technology such as a screen reader to convert the text materials to audio. Also, 
other participants mentioned that using a screen reader reduced their challenges when reading 
online materials, as they did not have to seek assistance in reading the materials. In addition, the 
finding showed that Word processing was the most helpful assistive technology because of its 
“Read Aloud” feature. These findings are consistent with the literature that identified text-to- 
speech assistive technology as the most beneficial to suit the individual needs of students with low 
vision (e.g., Hersh & Johnson, 2010; Fichton et al., 2009; Nees & Berry, 2013). In addition, this 
finding is tied to the “action and expression” UDL principle, which asserts that individuals with 
disabilities should get opportunities for independence through the use of assistive technologies as 
they help them in overcoming barriers in the educational environment (CAST, 2008). 


However, the findings of this study were not consistent with some literature on magnifiers 
as helpful assistive technology. The participants expressed that they have visual condition abilities 
to read the original document. Karen mentioned that magnifying or enlarging text were not helpful 
for her in online courses. Sarah mentioned that she avoids reading and she prefers using speech to 
text assistive technology to receive information through audio mode. According to this finding, 
magnifiers as the main method of providing accommodations were less useful than the ability to 
choose which forms of assistive technology were most beneficial for the online courses. 
Therefore, the participants needed to hear the information or conversations in online courses, so 
they did not need to use a magnifier but use speech to text features such as “Read Aloud” or screen 
reader. This finding evidence that online course designers should provide more accessible types 
of online materials to allow individuals to choose which type of assistive technology will work 
best for them such as text-to-speech. 


The third finding showed two aspects of design to increase accessibility and to facilitate 
reading for students with low vision: headings and color contrasting for the online content. The 
participants indicated that headings is helpful to direct their attention toward key concepts and 
facilitate navigation; however, the study did not reveal adequate headings styles for students with 
low vision. This finding is consistent with literature (e.g., Kearns et al., 2013) that recommends 
online course designers to design online materials with headings and use high-contrast colors, plain 
backgrounds, and scalable text for low vision or colorblind students as they allow them to skim 
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the page quickly. Headings allow students with low vision to locate the information more easily 
and grasp the main ideas of the text (e.g. Fichten et.al., 2009; Houston, 2018). 


Online materials with low contrast can be difficult to read for students with low vision, 
making color contrast necessary to improve accessibility (e.g. Houston, 2018). Sarah mentioned 
that she had difficulty reading the online PowerPoint slides because of the black and gray text font 
contrasted in a red background. She expressed this background color was not suitable for her 
visual condition. However, this study did not expand on color contrasted of online materials. 
Houston (2018) suggests avoiding some color combinations that are not easy to read for students 
with low vision, such as blue links on black backgrounds, red text on green backgrounds, or other 
combinations where contrast is not enough. Although, Houston (2018) study did not find a list of 
color combinations that can assure accessibility for students with low vision, his study suggests 
that materials in online courses should be presented using a dark font color contrasted with a pale 
background. 


Finally, UDL principles and literature (e.g. Kharade & Peese, 2012; Pittman & Heiselt, 
2014) address additional beneficial accessibility aspects for students with low vision, but the 
findings of this study showed some of these aspects were not applicable in participants’ online 
courses. Aspects that were not present in online courses for most of the participants in this study 
were closed captions on video media; a transcript of the video or audio presentation; visual analogs 
to represent emphasis and prosody (e.g., emoticons, symbols, or images); and text descriptors for 
any relevant image, graph, or chart. However, Sarah mentioned that added description for videos 
was helpful to understand the video contents. This finding supports UDL representation principle 
which suggests that presenting information in several formats increase accessibility. 


6.2. Wish List for Students with Low Vision in Online Courses 


The findings revealed that students with low vision need two additional accessibility design 
aspects: audio response and instructor video. Students with low vision can benefit from submitting 
their responses to discussion board as audio files. This would reduce time spent on formatting the 
answer, such as using a screen reader to double-check response and allow them to focus on content. 
This finding is consistent with (e.g. Ching & Hsu,2015) that addressed audio discussion modality 
in online courses; however, the literature does not address the needs of students with low vision. 

In addition, the participants expressed the need for videos in which their instructor presents the 
content. This finding is consistent with the literature (e.g. Choi & Johnson, 2015) that addressed 
the positive effects that instructor-recorded videos explaining content have on students as they 
improve students’ understanding and engagement with the materials. Relevant literature (Kim et 
al., 2019) suggests that audio representation of the content helped make curricula more accessible 
to students with low vision. Therefore, the findings of this study showed the need for inclusion of 
audio discussion modality and instructor’s audio representation of the content to reduce the 
challenge and enhance the learning of students with low vision. 


7. Recommendation, Implication and Conclusion 
The findings of this study serve as a foundation for future research on this topic. The 
literature review presents research on universal design for learning (UDL), especially as it relates 


to accessibility and assistive technology for students with low vision. This study will help 
instructors and online designers who might teach online courses or might want to improve online 
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courses. This study could primarily impact those students with low vision take online courses and 
face challenges, so they will have a better learning experience in online courses. Literature (e.g., 
Barnard-Brak & Sulak, 2010) found that students with invisible disabilities often are not 
comfortable disclosing their disabilities. The findings of this study also indicate online instructors 
should provide audio response to accommodate students with low vision in online courses. 

Based on the interpretations of the findings, this study focused on one type of visual 
impairment; however, a much broader future study can include other visual impairment types. In 
addition, this case study focused on the fully online courses, so future research can be replicated 
in blended courses, including face-to-face and online sessions of similar size and student 
population. The UDL theoretical framework can help future researchers replicate the study by 
focusing on specific principles of universal designs for learning (UDL) because such factors affect 
students with low vision engagement in online courses. Other research can expand the case study 
to examine students’ and/or professors’ perceptions of the accommodations and assistive 
technology for engaging students with low vision in online courses. In addition, this study was 
limited to students with low vision; therefore, future studies can expand this case study to examine 
instructors’ challenges when providing helpful accommodations for students with disabilities in 
online courses. 


The findings for this study showed students with low vision identified a screen reader as the 
most helpful assistive technology in online courses. Future studies can employ a quantitative 
approach to compare two groups of students with low vision to examine the effectiveness of using 
specific assistive technology in online courses. In addition, this study’s findings included the 
students with low vision preference for information delivery methods; future studies can employ 
quantitative methods to compare groups of students receiving different information delivery 
methods in online courses to understand the relationship between information delivery method and 
learning performance. 


The overall purpose of this qualitative case study was to explore the most helpful 
accessibility design and assistive technology for students with low vision in online courses. 
Individual interviews were conducted to obtain in-depth data. This study found that the most 
helpful aspects for online content accessibility for students with low vision are headings, color 
contrasting, and alternative formats for materials, such as Word documents, Rich Text formats 
(RTF) or Adobe accessible PDF files. Overall, this study reveals that online courses require the 
inclusion of more accommodations and better implementation of UDL principles to meet the needs 
of students with low vision. The consideration of the findings of this study may bring about 
significant understanding and renovation in the online courses design that will guarantee equal 
learning opportunities for students with low vision. Online designers, instructors and disability 
resource centers may benefit from this study as the findings can guide their decisions on providing 
support to students with low vision. 
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Appendix A 
Semi-Structured Guiding Interview 


When were you diagnosed with low-vision? 

In which educational level you did recognize you needed more support and 
accommodations from the school or teachers/instructors? 

Do you have other family members who have the same or a similar condition? 

Do you learn from them? Or did you teach them how to deal with low vision in academic 
setting? 


As a student with low-vision, do you prefer online or face-to-face courses? 


Face-to-face course 
Online course 
Hybrid/Blended course 
All types of courses 


a. Why do you prefer that type of course? 


What are the information delivery methods (text such as pdf or word document, audio, 
video) that you find to be most beneficial with regard to your learning in the past online 
courses? 
a. How did you use those methods of delivering information to help your personal 
learning preferences? 
b. What currently unavailable methods of providing the information would you like 
to become available in the future? 


How has accessibility and assistive technology helped you to overcome challenges in 
your online courses? What were these challenges? 


Which types of accommodations and assistive technologies could contribute better to 
your engagement, participation and learning of the content of your online courses? 


According to your experience, what accessibility accommodations in online courses were 
helpful to you and how were they helpful ( to navigate the online courses, to better 
understand online instruction and/or to complete online activities?? 


a. What accessibility design did you need in online courses but did not get to help 
you understand the material? 
. What kind of visual information were you looking for in online courses? 
c. What kind of auditory information were you looking for in online courses? 


What is your wish list regarding accessibility accommodations you’d prefer in an online 


course? Can you describe a specific experience where you felt like you didn’t have access 
to services or accommodations that you thought would be helpful in your education? 
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7. Is there anything else you would like the researchers to know about your online course 
experience regarding your low-vision? 


8. What question should I have asked, but didn’t? 
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ABSTRACT 


Social media plays an important role in the interaction and communication between 
home, especially culturally diverse families, and schools. This qualitative multiple case study 
aimed at exploring how international students who are also parents respond to the use of 
social media in communicating with the school and teachers in the US. Participants responded to 
the question - How can social media tools (Or new technologies) be used to engage parents, 
families, and communities in education? Five participants, who were students from a US 
university at the time of the data collection, volunteered to be interviewed. The participants were 
from three regions — Asia, North African, and the Middle East. The study found out that digital 
used by the school to engage culturally diverse parents assisted the parents to familiarize 
themselves with local cultures. Also, parents who used social media community platforms such 
as WhatsApp commented that the tool has helped them to build community among parents and 
suggested that schools should use such mediums to help build a community for parents. 


Keywords: parental involvement/engagement, community building, international students 
/parents, culturally diverse parents/families/communities, social media, education technology, 
digital tools 

INTRODUCTION 


Technology has been a powerful tool for transforming social interaction and assisting to 
strengthen the relationship between teachers and parents (Graham-Clay, 2005; Olmstead, 2013). 
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Thus, it has helped to affirm and advance relationships between educators, students, and parents. 
Parents working schedule impacts the number of time parents could visit schools. Some parents 
have busy working schedules and some also work double shifts (Alzaid, et al. 2020). As a 
result, technology as a medium can be used by schools to inform parents about students’ 
achievement and behavior, and school activities. In addition to parents’ working schedule,-there 
may also be several unforeseen contingencies that could restrict direct teacher and parents’ 
physical interactions. This warrant the need to explore other avenues to maintain teacher-parent- 
communication relationship. It, therefore, has become relevant that teachers engage 

parents through using technology, especially, social media tools. 


Parents and family involvement in education has a vital role in children's education 
(Epstein, 1985; Roekel, 2008). In addition, the community as a whole plays an essential role in 
the growth and development of youth. School, family, and community partnerships are related to 
student achievement and school success (Roekel, 2008). Epstein (1992) describes six types of 
involvement: parenting, communicating, volunteering, learning at home, decision making, and 
collaborating with the community. This shows the multiple roles parents play in their child’s 
educations. Parents play a vital role in students’ learning and such roles are confined to the 
home. As Epstein indicated, with continuous communication, teachers and parents can better 
position themselves to understand students’ and experience so that they can provide learning 
experiences that support students’ learning. 

Cibils (2017, p.55), on the other hand, revealed that “what accounts for parental 
involvement is unilaterally defined by the school and limited to those behaviors and attitudes 
which reflect strict alignment with school policies and decisions”. There are, therefore, certain 
cultures that do not conform to the mainstream culture that influences school policies. As a 
result, such behaviors are mostly sidelined by the school structures either consciously or 
unconsciously. Therefore, since parental characteristics influence how they interact with the 
school and among themselves (Civil, Bratton, & Quintos, 2000; Epstein & Salinas, 2004; Erol & 
Turhan, 2018; Yamamoto, 2015), it becomes relevant to explore how schools promote parental 
social networking between linguistically diverse parents. 


As stakeholders of education are considering the use of social media tools to engage 
parents (Willis & Exley, 2018), it becomes pertinent for researchers to explore the attitudes and 
perceptions of parents from different cultures other than the mainstream culture (Alzaid, et al. 
2020). Diverse parents’ perception of social media adoption for community building between 
parents and teachers will influence how teachers adopt tools to engage parents. This because the 
heterogeneity of parents’ culture impacts how they interact in every social environment (Gentina, 
& Singh, 2015). Consequently, teachers’ knowledge of parents’ attitudes and perceptions will 
help teachers to assist parents to develop a positive attitude toward social media and use 
emerging technology to a sense of community. The purpose of this article is to share the results 
of a qualitative multiple case study aimed at exploring how international students who are also 
parents respond to the use of social media in communicating with the school and teachers in the 
US. 


LITERATURE REVIEW 


This section gives an overview of research on parental involvement and parents to 
parents to parent interaction. It also highlights how communication technology tools have been 
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used by schools to engage parents and the challenges associated with using communication tools 
to engage parents. 


Parental Involvement, Parent-Teacher Communication and Parent Social Networks 
Parental involvement occurs in many forms. Parents engage in students’ home learning, 
participate in school events, and volunteer in their children’s school. Parents also communicate 
with teachers about students’ learning (Alzaid, et al. 2020; Epstein, 1992, 2011). Hence, parental 
involvement transcends beyond teachers engaging directly in students’ learning at home to 
include the relationship they maintain with teachers. Communication between parents and 
teachers has been proven to improve students’ academic achievement (Epstein, 2011; Hornby & 
Blackwell, 2018). In the view of Vijaya (2017), parent-teacher communication is vital in parental 
involvement. It should not be seen as an ephemeral interaction between teachers and parents 
about students’ performance, but long-lasting relationships teachers and parents create with the 
view of promoting life-long learning for students. 


In addition, research indicates that parents’ involvement in their children's education 
could be strengthened by social networks that are created among parents (Curry & Holter, 
2019; Ee, 2017). That is, when opportunities are created for parents to interact among 
themselves, they can support each other and improve their involvement in students’ education 
both at home and at school. Supporting parent to parent interaction leads to a sense of 
community for parents (Alzaid, et al. 2020). Parents are capable of supporting each other to 
promote their involvement in their children's education. Parents discuss among themselves 
students learning, school activities, and challenges they face engaging in their students' learning. 
It is against this backdrop that schools should support parents by developing communication 
tools that bring parents together. 


Also, parents 'social networks evolve easily along the lines of parents' demographic 
characteristics (Leo, 2010). Thus, parents easily build networks with other parents who share the 
same characteristics and interests. However, bringing parents together can be challenging in 
elementary schools. Schools lack efficient strategies that could bring linguistically diverse 
parents together to build a sense of community or parent social networks (Curry & Holter, 

2019; Ee, 2017). That is, parents’ involvement in their children’s learning is strengthened by 
social networks that are created among parents, however, the creation of these social networks is 
made possible when schools provide opportunities for parents to interact among themselves 
(Curry & Holter, 2019). 


Using Technology to Improve Teacher-Parents Relationship 


Technology has improved communication between parents, communities, and schools in 
several ways. Graham-Clay (2015) explained that classroom phones and voice mail, video 
technology, radio announcement, and school websites are ways teachers can utilize to engage 
parents. Similarly, Hendley (2002) recounted that online communication is effective in 
improving parent-teacher communication. Analyzing a schools’ webpage, Hendley hinted that 
such that webpages engaged parents in school activities. Bahceci (2019) also realized that the 
use of a digital tool, Class Dojo, by schools to engage parents increased the rate at which 
parents inquired about contents their children were studying at school. Thus, using technology, 
teachers will be able to engage parents in schools’ activities. Parents, therefore, need not be in 
the physical environment of the school; they can still inquire about schools’ activities and 
participate remotely in some school activity. According to Beecher and Buzhardt 
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(2016), Innovative tools and programs that utilize mobile technology possess the capability of 
delivering content to parents and teachers. Thus, teachers and parents should be 

actively engaging with mobile technology to increase their enthusiasm to adopt innovative 
technology. 


Thompson, et al., (2015) noted that the use of smartphones could augment 
communication through the use of social media to increase the modes of parent-teacher 
communication. They found that most parents preferred emails since they can reply at their own 
convenience time. Again, the proliferation of smartphones will enable parents to easily access 
emails from teachers. They also note that texting also works well for getting quick feedback from 
parents. 


Olmstead (2013) aimed at determining how emerging technologies facilitate efficient 
parents-teachers communication and parent involvement. The study was conducted through the 
lenses of social constructivism and the social cognitive theory of self-efficacy. Olmstead used 
semi-structured interviews to collect the data. The study revealed that technology assists in 
keeping parents informed about how students are performing. Also, the study revealed that 
"majority of parents were interested in receiving information about their child through the use of 
text messaging, but most of the teachers were not willing to use this type of communication 
p.36." This indicates that there are some constraints in using technology to engage parents. 
Teachers' willingness to use a particular tool or strategy plays a role. 


Challenges to The Use of Technology in Teacher-Parent Relationship 


As schools and teachers use technology to improve their communication with parents, 
they also experience several challenges. Patrikakou (2016) indicated that the digital divide 
between the higher income and lower-income families seems to close due to the expansion of 
mobile technology and the declining cost of mobile devices. Also, the call for national broadband 
technology hints at a future where families could access the internet and social interaction will be 
continuously be altered. Graham-Clay (2005) reiterated that teachers should be aware of families 
that may not have access to technology so that they do not marginalize them. For instance, 
Hendley (2002) also ensured that parents who do not have access to computers were provided 
with weekly newsletters. 


Patrikakou (2016) concluded by stating that everyone needs to be well versed in the use 
of technology, and schools should be able to assist parents to navigate the use of technology and 
media to be well equipped to get involved. That is when parents realize the usefulness of mobile 
devices and acknowledge their ease of use, schools would be able to mitigate barriers parents 
experience when it comes to getting involved in students' education. This mediating process, in 
the form of educating parents on technology use, would parent to build resilience. Also, Graham- 
Clay (2005) presented opportunities that include technologies that are available for teachers to 
implement to increase parents! involvement in education. Graham-Clay stressed that teachers 
need to continue the development of their skills to improve their interaction with their parents. 


Ho, Hung, and Chen (2013) further showed that teachers’ attitude influences their 
adoption of communication tools. They investigated teachers' attitudes towards adopting mobile 
phone texts as a medium for parent-teacher communication. They concluded that "perceived 
usefulness and perceived ease of use have direct positive relationships with attitude, and attitude 
further influences behavior intention directly” (p.113). The study suggested that attitude 
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should be seen as a mediator between perceived usefulness and behavior intention. Schools 
should therefore factor in parents’ attitude as they design social media tools to engage parents. 


THEORETICAL FRAMEWORK 


To ascertain the perspectives, beliefs, and strategies ISP who have been using educational 
technology or digital tools for CB, strengthening PTP and to be involved in their children’s 
education in U.S, the study will use Epstein’s six types of parental involvement. According 
Epstein (1985), schools need to provide opportunity to assist parents in the area of parenting, 
communicating, volunteering, learning at home, decision making, and collaborating with 
community. Epstein focused on the need for collaboration between the home, community and 
school to assist parents in the various types of parental involvement. Therefore as “Technology 
can decrease the sense of fear that educators and families often experience as they start to think 
about increasing collaborative activities, or the isolation that some schools feel as they begin to 
develop a program of partnerships” (Simon et al. 1998, p. 268), the study will use Epstein’s six 
types of parental involvement to examine how educational technology has assisted ISP in 
parenting, communicating, volunteering, learning at home, decision making, and collaborating 
with community by exploring their beliefs, perspective and strategies. 


RESEARCH DESIGN 


The research design for the study was a qualitative multiple case study with the 
overarching research question - How can social media tools (Or new technologies) be used to 
engage parents, families, and communities in education? Participants who responded to semi- 
structured interview questions were international students who couple as parents with their 
children either in the elementary school or high school in a U.S. Southwest borderland. In all, 
five (5) international students from North Africa, the Middle East, and Asia participated in this 
study. The participants were made up of four males and one female. 


Data was collected through semi-structured interviews. The semi-structured interview 
was by the researchers to allow for conversations, between the interviewer and the 
interviewee, that are impulsive and include in-depth answers (Ryan et. al., 2009). Interview from 
each case was transcribed and analyzed for relevant themes. 


To identify contrasts and possible similarities across the case, this study used 
comparative analysis (Parra, 2010; Miles & Huberman, 1994). Comparative analysis also aided 
in deepening researchers’ understanding and their interpretation of responses in relation to the 
research question that was asked. Comparing responses helped the researchers to address validity 
and reliability of the questions (Miles & Huberman, 1994). Optional member checking was used 
for the trustworthiness of the study, and participants were sent the case study write-ups to check 
for accuracy (Guba, 1981; Lincoln, 1995). It must be noted the data was collected before the 
COVID-19 pandemic. However, the analysis of the data occurred during the pandemic. As a 
result, the researchers used web conference tools and shared document tools, Zoom, and 
Microsoft online respectively, to continue working on the research paper as a team. As the 
researchers were analyzing the data, they also took into consideration the effect of the pandemic 
on technology/digital use. 
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FINDINGS 


Responding to how social media tools could be used to engage parents, families, and 
communities in education, all the participants, affirmed that social media tools are effective 
communication tools schools can employ to assist diverse parents. The study found out that the 
international students, who have their children in the US elementary and high school, perceive 
that social media could assist them to build a community among parents and with the school. 
Also, Social media tools could help them familiarize themselves with the local culture and 
further make it easier to seek academic support for children from other parents and the school. 


Community Building 


The study unearthed that social media tools fostered building a community between 
parents and schools. The results indicated that if parents are put on one platform and encouraged 
by the schools, they will interact with each other. Most of the participants, who noted that they 
belong to parents' groups on WhatsApp (not related to school parents’ groups), elucidated that the 
tool assists them to communicate among themselves, and also share information. They 
recommended that schools could capitalize on the opportunity to bring parents together. Social 
media tools, therefore, serve as an opportunity for improved parent-teacher 
communication. These findings align with findings from Bahceci (2019) and Thompson et al. 
(2015) that the use of digital tools by schools to engage parents increased the rate at which 
parents communicated with the schools that the use of smartphones could augment 
communication between parents and the school through the use of social media. 


Abdullah from the Middle East, for instance, responded that using social media is helpful 
though his children’s school was not using it. He stated, “J think social media has a good 
advantage to communicate with school society and parents". Using social media, Abdullah felt 
he could communicate with parents, especially parents from his home country he had never 
met. 


He also stated, “J know there are some Arab parents in school where my son attends but 
don't know who they are because there's no way to communicate with them”. Hussain, who is 
from the Middle East also revealed that he used a tool like WhatsApp with friends for social 
interactions not for the school. However, though he specified that he does not regularly use 
WhatsApp for interaction, a social media tool for parents to interact among themselves will be 
helpful. He stated, “J don’t have time to interact more with these social media tools...to 
communicate with parents, sometimes you need [An opportunity for parents to interact among 
themselves using social will be helpfull” 


Fatima who is from North African recounted that she uses a group chat on the WhatsApp 
platform with other parents, and it helped them to build a parents’ community. She also noted 
that it has helped them to make it easier to build a community among themselves to support their 
children’s education. Similarly, Adnan commented that social media tools are capable of 
enhancing communication between schools and families. He made this claim in reference to 
Facebook messenger since he is familiar with this tool. Adnan stated, “But now it's the time of 
technology...using just the Messenger group, even if it's not the best communication channel...if 
you want to enhance that communication, it is helpful.” 
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Providing Academic Support to The Children 


The study also found-that bringing parents together and using social media could 
facilitate the provision of academic support for children. Thus, when parents come together, 
they are able to support each other to find solutions to academic problems. 

This demonstrated that the provision of academic support will not be the sole responsibility of 
the school. Also, as international students, the participants showed that academic support their 
children is needed if their children are to survive in the US school system. However, with access 
to digital tools and using social media, it will be easier for parents to build networks and support 
each other. Similar research also realized that parents, collaborating among themselves to build a 
network, support themselves and it improve their involvement in the children’s education. 

Both Abdullah and Fatima expounded that social media tools had helped build a 
community where parents could acquire academic support for their parents. In Abdullah words 
when explaining how social media groups could provide support for parents stated: 

Through my experience with using WhatsApp group, our community 
can share some experience related to academic and they help each 
other when someone need help in an academic matter like homework, 
and discussion about some classes and share advice. Always they help 
each other when they face some problems. 

Similar Fatima shared an experience she has with using a WhatsApp group. In Fatima words, 
[We] created WhatsApp group to share some material that help us as 
parents to help our children. We do not have any experience, so we 
had to find out by [our] own. We needed to look for resources and buy 
books for our children’s exam. [This is] one-way schools can use 
technology [to help people] who are planning to take the ACT 
test. They [schools] can create groups [digital tools] for parents or for 
students to communicate and share resources and even ideas. 

The experienced they shared indicated schools could assist parents to form social 
networks using social tools. The goal for such an endeavor will be assisting parents to share 
academic resources among themselves. This will however assist new parents or parents from 
different cultures to makes inquiries about where they can get educational resources. Also, it will 
make it easy for diverse parents to seek help not only from the school or teachers but also from 
other parents. Social media tools will therefore make it easier for parents and teachers to find and 
share resources. 


Familiarizing Parents with The Local Culture 


The study also revealed that the participants acknowledge that if schools use social media 
to engage families and teachers, it will assist parents to get acquainted with local culture and 
share their culture. Some of the participants indicated that when teachers or schools send in-class 
activities through digital tools, it helps them to learn about the cultures and traditional 
celebrations in the United States. Most parents admitted they do not know much about the local 
culture and celebrations their children participate in while at school. However, when an 
opportunity was created for teachers to take pictures and video the activities to a school platform, 
parents were able to find out more about activities in the US. Thus, if schools use social media 
tools where they can share activities in school with parents, they will be well informed about the 
event in their new environment. 
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Amir from Asia shared that using social media tools will assist him to know many 
cultures. Though he may not physically present in the school environment, the images teacher 
will post will give him an idea of what is transpiring in the school. Amir explained that most of 
the things happening in the classroom are based on the local culture and they, are parents, should 
be able to experience. Amir stated: 


Valentine's Day and so many other cultural things going on in school. 
What I find interesting is I get to know more about celebration and so 
many things which I have never, ever heard. One thing gives a lot of 
local cultural flavor. I've been here for five years but I don't know 
many cultural things here. But the class Dojo [school adopted 
communication tool] helped me because they celebrate different 
cultural events in the classrooms and immediately share for us to see. 


Adnan on the other hand noted that the school’s communication tools have helped them 
to know about the local culture and also comprehend the school’s environment. Adnan on the 
other hand noted that the school’s communication tools have helped them to know about the 
local culture and also comprehend the school's environment. He stated, “They [digital tools] 
helped us at least. Gather our thoughts and understand the school environment, because, without 
those communication tools, we wouldn't know really nothing about this [local] culture’. 


CONCLUSION 


The research study shows that the use of social media networks and modern digital tools 
will enable parents not only to communicate with teachers and the school but also with other 
parents’ community. Social media communication thereby could help build a community among 
parents as well as it could expanding parent engagement to include culturally diverse parents 
which can benefit families, schools, communities, and for the futures of individual 
children. Also, throughout this study, it was made clear that diverse parents benefited from using 
social media as it assisted them to learn about the interior cultures, traditions, and annual feasts 
that are customary in the United States of America. This study also showed that the use of social 
communication among parents has become a means for parents to solicit academic support for 
their children. Parents could use social media platforms to communicate among themselves to 
show solidarity and cooperation to obtain academic support for their children. 


Also, earlier indicated, the analyses of the data took place during the COVID-19 
pandemic. Researchers further realized the study could be relevant as physical contact between 
humans has been restricted. Thus, the increase in COVID-19 should serve as a hint to school to 
find possible means of improving and maintaining teacher-parent communication. A pandemic 
should not further widen the gap between parents and schools. It is against this that schools need 
to effectively utilize emerging technology to stay in contact with parents and build a sense of 
community using social media tools. Schools should also assist parents to realize the usefulness 
of using communication technology tools to connect with the schools. 

This study included only five participants and it will be relevant to include more 
participants in future studies. Also, the pandemic became an obstacle since the study made 
provision for only face-to-face interviews as a result, not all targeted participants were accessed 
by the researcher. 
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According to the Best Colleges.com (2018) survey of online programs, fully online 
programs have increased by approximately 31% between 2016 and 2017. Prior to the 2020 
Pandemic at least 25% of all Minnesota State University (2019) courses were online system 
wide. Due to the impact of COVID 19, the move to virtual learning has been quick and often 
without the professional development and other infrastructure supports historically provided 
faculty in making the transition to online instruction. This lack of appropriate support has been 
one of the reasons the past year’s instruction has often been referred to “remote learning” as 
opposed to the traditional terms of “online” or “virtual” learning (Craig, 2020). This practice has 
helped to accentuate the need for proper pedagogical professional development for instructors to 
be fully prepared to teach in an online environment. According to Lederman (2019), only about 
half of professors are comfortable in an online environment. Meanwhile, only 26% of college 
students are satisfied with an entirely virtual classroom experience (Burke, 2020). During the 
spring of 2020, in some reported cases, at least 40% of students were not engaged in their remote 
or virtual courses (Quesada, 2020). 


As the potential for remote or virtual learning appears to be pushing into the spring of 
2021 (Liesman, 2020; Natanson. & Strauss, 2020), it is essential to ensure instructors have the 
ability to engage their online students. Students and faculty are both concerned about ensuring 
adequate engagement within online courses. Therefore, as a project with the scope of a 
scholarship of teaching and learning grant at a midwestern university, the author and several 
faculty colleagues looked to determine the impact on student engagement of three key 
approaches to improving student engagement in online asynchronous graduate courses. The first 
was the potential impact of providing student choice within the required discussion threads of the 
course. The second was the impact of requiring students to complete an introductory video for a 
course. A third was adding a voluntary synchronous study session to an asynchronous online 
course to engage students more fully. 
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Research Questions 


During the 2019-2020 academic year, several faculty in an online asynchronous graduate 
program decided to try to determine effective ways to improve student engagement in the 
courses. There were three primary research questions: 


Research Question 1: Did student choice in asynchronous online discussion boards 
generate more student engagement? 


Research Question 2: Did the addition of a video-based introduction requirement increase 
student engagement? 


Research Question 3: Did the inclusion of a voluntary weekly synchronous study session 
improve student engagement in an otherwise asynchronous course? 


This research study was limited to fully online asynchronous courses as part of a graduate 
program at a midwestern university leading to a degree in either special education or educational 
administration. 


Background 


One of the reasons students are concerned about online courses is the lack of personal 
engagement with the instructor and their peers. At the same time, potential students are worried 
about the potential lack of quality in online programs and the perceived lack of interaction and 
community among classmates and with the professor. (Best Colleges, 2018; Burke, 2020). 
Student engagement remains a significant concern for online instructors. 


A secondary factor considered was whether or not students participated in a video 
introduction with their peers at the beginning of a course. A recent metanalysis (Ma, 2019) 
considered video introductions as a key determining factor of student engagement. Similarly, the 
University of Phoenix’s College of Doctoral Studies (2020) stated 85% of students felt all online 
courses should include a welcome video and 95% of students felt an instructor’s welcome video 
encouraged them to reach out to their instructors. 


Methodology 


This exploratory mixed method study reviewed the impact of three particular 
instructional practices on student engagement within asynchronous online course discussions, the 
provision of choice to students within the discussion prompts (RQ1), the use of video 
introductions within the course (RQ2), and in two courses, the addition of a voluntary 
synchronous study session (RQ3). The methodology consisted of an ex-post facto analysis of the 
student responses to weekly discussion prompts in multiple sections of asynchronous online 


50 


graduate courses and a review of the data available from within the CANVAS LMS system. 
Some of the discussion prompts provided students choice in what to discuss and others did not 
provide for student choice. The initial review was an analysis of the number of posts per student 
and the overall word counts of those responses. 


The fact of whether or not students were asked to participate in a video introduction to 
their classmates was considered as another potential factor in developing student engagement as 
well in conducting a similar analysis (RQ2). The research then considered the overall grades 
earned by students for the discussion posts, the length of their posts, and the number of posts the 
students made. Increases in those areas were considered to be evidence of increased engagement 
by the student when the requirements within the course was not otherwise modified. 


In addressing the third research question, the researcher reviewed the system end of 
course surveys to identify the potential impact of the synchronous study sessions. Additionally, 
the students from the Summer 2020 course sessions were provided the opportunity to complete a 
survey to respond to the impact of the instructional changes intended to increase student 
engagement in their courses. The non-demographic questions from the survey queried students 
about: 


Do you feel you are learning enough in the course at this time? 

Do you feel the optional Zoom sessions are helpful? 

Do you think I am participating or moderating the discussion questions? 

Do you prefer to have a couple of questions to select from in the discussion threads or 


hee ee a 


would you prefer all students to answer the exact same discussion prompts? 

5. Do you feel you are getting adequate feedback on your assignments? 

6. Do you find the video introductions helpful as a way to get to know each other? 

7. If we were able to allow for greater differentiation in the course, by allowing students to 
choose one or more modules to complete would you be in favor of that approach? 

8. What additional things could I do to improve student engagement within my online 
courses? 


Prior to employing the new options for students, students in courses without choice within 
the discussion prompts were asked if they felt providing choice within the discussion prompts 
was a good idea. The results were initially split. The question, “Would you prefer to have a 
couple of questions to select from or continue to have everyone in the class respond to the same 
questions?” How respondents answered is shown below in Figure 1. 
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Figure 1 
Percentage of students who wanted to implement choice in discussion threads. 


23 
46 
31 


Choice would be good, 46% 
Keep everyone focused on the same question, 31% 


It doesn't matter to me, 23% 


The discussion threads were then divided into those offering choice and those without student 
choice. They were compared by number of student posts, the length of posts, and the overall 
grades obtained by students for each approach to the discussion section of the course. A similar 
review was conduced based upon whether or not the course required an introductory video. The 
survey results were relied upon for the impact of the synchronous study sessions as they were 
only implemented by one faculty member in two courses. Faculty who had implemented choice 
or video introductions were asked for feedback as well as to the impact of the new facets of their 
courses. 


Results 


The Impact of Choice within Discussion Threads 


The goal of adding choice to the discussion threads was to allow students to have a 
greater voice in determining the direction of the discussions and therefore encourage student 
engagement. After the initial pilot survey about choice, one initial course was offered with 
choice within the discussion threads. At the end of the course, they were given a similar survey 
as to the one mentioned in Figure 1. At that point, 94% of the students identified that they 
preferred choice and 6% stated no preference. None of the students wished to return to a system 
without choice. 


Choice was then implemented within approximately half of the course load of the faculty 
involved. A simple analysis of the discussion threads from the courses showed that students in 
the courses where they were given choice in their discussion prompts wrote posts that were 
approximately five percent longer than those without choice. Overall, average student grades for 
the discussion threads were approximately 96% without choice and 98% when they were offered 
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choice. The specific numbers regarding the length of posts are shown in Figure 2. The basic 
discussion rubric used in all of the courses involved expected a minimum of 150 words per post. 


Figure 2: 


Scope of Discussion Prompts with and without Choice 


Average Word Count per Post 


Grane 225.98 
No 
Choice 213.15 
205 210 215 220 225 230 


When offered the opportunity to voice their comments about choice in the discussion 
portion of their courses, 61% percent of the respondents preferred choice. 16% of the 
respondents would have preferred to have students focus on a single set of questions without 
choice. The other 22% did not have a preference. This was higher than the percentage of students 
who wanted choice prior to being exposed to choice within a course. Only one respondent added 
a specific comment about the discussion threads offering: 


To add excitement I think laughter is always good...maybe some "what is your 
favorite.....to do outside of school?" or "what is your favorite meal to grill outside?" Just to get 
us talking about something that we could find common ground with. It could build cohesiveness 
in cohorts. 


When reviewing the number of discussion posts in courses with and without choice, there was no 
effective difference between the number of posts made by students. Faculty felt that the choice 
encouraged both student engagement and the development of higher-level thinking skills. 


The Impact of Requiring Video Introductions 


Asking students to post an introductory video appears to have had a major impact on 
student engagement. Students appear to have responded positively to the implementation of 
introductory videos. Sixty-nine percent of the respondents felt that the introductory videos were 
helpful for students get to know each other. The other 31% didn’t feel the video introductions 
were helpful. 
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Students were provided a prompt to respond to in order to guide their response within the 
introductory video. An example is below: 


Getting to Know You 


o An important part of learning online is creating a learning community and a sense of 
interaction with the other members of the class. The introductory video and your profile 
are a great first step towards that goal. 


o Create a brief 2-3 minute video introduction that tells the class more about who you are. 
In your introduction, you could include a little about yourself, your background and work 
experience, why you are enrolled in this course, and after looking at the syllabus 
something you are excited about. 


As illustrated in Figure 3, students in courses with a video introduction requirement 
posted 33% more than students without the requirement. Additionally, the posts were slightly 
longer on average as well. Initially, several of the course had the introductions completed 
through FlipGrid, a third-party service. Then all the introductory videos were managed through 
the CANVAS LMS in order to save students having to set up an additional account or potentially 
learn another system. It does not appear to have made a difference to student engagement as to 
whether the videos were managed through FlipGrid or within the CANVAS LMS itself. 
However, among students who completed the Summer 2020 survey, all respondents rated their 
technology skills as average or above average (the highest rating), so working with students with 
lower levels of digital literacy might appreciate not having to learn an additional technology tool. 


Figure 3 


Number of Posts Per Week Based upon Video Introduction Requirement 


Video introduction 5.42 f 
Video introduction outside of the LMS 5.37 f 
No video introduction 3.93 f 
0 1 2 3 4 5 6 


Student survey responses were uniformly positive. Some specific comments regarding the 
introductory videos included: 


The video introductions allow more of our personalities to show through. 


L, for one, am very much enjoying the video introductions! 
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I like when we have options to do short video discussion threads. It does make the 
class more personal. 


Faculty responding to a request for insights on using introductory videos stated using helped to 
engage students and to help the faculty get to know the students as individuals. Some faculty 
stated they returned to the videos throughout the course as a way to remind themselves of the 
students’ backgrounds and interests. 


The Impact of Synchronous Study Session in an Asynchronous Course 


Due to the impact of COVID 19, a voluntary study session was added to assist students in 
coping with the isolation of the pandemic. The session was held in the evening on the same day 
or the week throughout the course. It was clearly articulated that the sessions were voluntary and 
the sessions were recorded and the recorded sessions were posted into the LMS for students who 
wished to review them or who had been unable to attend the live session. One hundred percent of 
the students exposed to the study sessions responded that they felt they were helpful. In general, 
at least two-thirds of the students participated in each of the study sessions. Several sessions had 
100% attendance. Specific responses about the study sessions included: 


These meetings help me connect with you and my classmates. These connections 
motivate me to extend myself on the discussion board posts. The meetings are 
also good springboards for the weekly writing assignments. 


Although online independent study really fits my career/lifestyle with my family at 
this time, nothing can replace the face-to-face conversation and thought process 
you get when talking outloud. I really appreciate these meetings and having them 
to "add" to what we are discussing, reading, researching, and experiencing. 


Very helpful most of the time. Great to connect with others, hear their 
questions...etc. 


I liked the balance of online work and the optional weekly face to face [sic] 
meetings. 


This class was the best online learning experience I have had ... ... [the 
instructor] offered short weekly Zoom meetings. These were very helpful in 
guiding my learning and motivating me. They also helped create a sense of 
community with my peers. 


Students who did not have the opportunity to participate in the synchronous study sessions were 
asked it they would like access to such sessions. Fifty-two percent of the respondents said they 
would prefer not to have such sessions. That did not align with the fact 78% of the same group of 
respondents said if the time wasn’t convenient to them, they would utilize a recording of the 
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session. A second brief poll of students was conducted in the Fall of 2020 with current students 
and 93% found the synchronous sessions to be valuable. 


Some Additional Insights 


During the review of the discussion posts by students and other LMS data, two additional 
insights were drawn from the LMS data. The first was that there was a moderately strong 
positive correlation between the number of discussion posts a student made and their final course 
grade as shown in Figure 4. A second insight was that simple access to the course in the LMS 
was not enough to improve student performance. The correlation coefficient for page views v. 
final grade was r = 0.3632 compared to the r = 0.6084 for the number of discussion posts v. final 
grade. Simply viewing course materials was not as effective in raising student performance as 
active engagement in the discussion threads. 


Figure 4 
Final Grade v. No. of Discussion Posts 
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Conclusions 


This study was based upon the review of graduate level education courses delivered in a 
asynchronous online format .Students were not in cohorts and the course scheduling process 
generated a range of students within each course from first time graduate students through 
students in their final course. Courses were identified as either providing for choice in discussion 
prompts or not allowing choice. Each course was generally structured to include weekly 
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discussion prompts as part of the course design and those discussions accounted for 20% of the 
student’s final grade. 


Recommendations for Practitioners 


Students tended to perform at a higher level of engagement when they had choice within 
the discussion threads. The students provided about five percent longer posts when they had 
choice in which discussion prompts to respond to. The students who had the opportunity for 
choice lost about half the amount of discussion points that students without choice lost during the 
length of the course. Therefore, it is recommended to allow students choice in the discussion 
prompts to which they are asked to respond. 


Having students and the instructor create introductory videos at the beginning of the 
course is another way to improve student engagement. Students in courses requiring videos 
posted about a third more than students without such a requirement. Open responses similarly 
show students felt the introductions were helpful in building community and engaging students. 
Therefore, asking students to complete an introductory video within the LMS, if possible, is a 
good way to improve student engagement within online courses. 


Adding synchronous study sessions is another recommendation after reviewing the 
responses from students about the impact of the sessions and 100% of the students responded 
that the sessions were useful. It is important to share with students that active participation in the 
discussion threads is a positive method for improving student performance. Students sometimes 
think of the discussion threads as busywork and not a core component of the coursework. 


Recommendations for Further Research 


This study was conducted among graduate students in educational graduate programs. It 
would be helpful to replicate the three facets of the study among undergraduate students. 
Additionally a study utilizing secondary students, who are also experiencing a high level of 
online instruction and reportedly low levels of engagement in many areas, would be helpful as 
online learning is likely to remain with a lager presence in that realm of the foreseeable future. 


Another way to build on this study would be to complete a content analysis of the 
discussion posts themselves to confirm that choice does encourage higher levels of thinking 
skills as defined by Bloom’s Taxonomy (Anderson, L. W., & Krathwohl, D. R., 2001). 


As the initial study was conducted in a rather homogeneously white rural area of the 
Midwest, testing the recommendations made in areas with more heterogeneous demographics 
may potentially provide additional insights. 
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Abstract 


In an attempt to break a micro-culture of shallow learning associated with problem- 
solving, this study describes the impact of two complementary instructional deep-learning 
strategies, productive failure and instructor-designed concept maps on pharmacy students’ 
perceptions and beliefs. The analysis of a proposed path model indicated that students’ perceived 
impact of the two instructional strategies were significant predictors of self-efficacy. Future 
research will focus on the impact student-generated concept maps for transfer problems 
associated productive failure. 


Motivation and Objective of the Study 


Pharmacy students are expected to assume diverse roles that extend beyond the 
traditional roles of dispensing medications and medication management for patients. They are 
increasingly expected to analyze new information and use the conclusions drawn to solve 
complex problems. Current ACPE pharmacy accreditation standards emphasize problem-solving 
competencies needed to integrate knowledge from foundational sciences (Medina et al., 2013). 
The instructional strategy commonly used in fundamental and applied sciences is scaffolding 
through worked examples (e.g. Jonassen, 2004, 2011). One of the major shortcomings of this 
strategy is its tendency to prompt learners to focus on the procedural aspects of the problem 
solving. That is, students are often missing the conceptual integration of foundational knowledge 
that experts convey as they work though problems (Darabi et al., 2007; van Gog et al., 2015). 

This shortcoming become even more critical, especially as pre-pharmacy students create 
a micro-culture of shallow learning. As these students move into the entry level pharmacy 
courses, some of their gaps in knowledge pose challenges that could hinder their academic 
performance. The major objective of this study was to analyze to what degree instructional 
strategies focused on stimulating deep conceptual understanding through the use of concept 
mapping integrated with productive failure in-class strategies can alter the above described 
micro-culture of shallow learning. 


Instructional Intervention 


“Pharmaceutics I” is a foundation course in the Pharmacy Doctorate (PharmD) program 
that strives to bridge the foundational chemistry knowledge form pre-pharm curricula and the 
applicative clinical skills by building strong analytical and problem-solving skills. The instructor 
found that for some topics for which students consistently performed below expectations and 
identified as one major reason the fact that students failed to make the required connection 
between the conceptual aspects of chemical equations and the algebraic equations used to model 
them. 
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For a couple of semesters, the instructor worked with an instructional designer to adapt 
and implement a productive failure instructional strategy (Kapur, 2008, 2010, 2013). Using a 
design-based research methodology (Cernusca & Ionas, 2014) with two iterative DBR cycles, the 
researchers found that students in the treatment cohort scored significantly higher on the exam 
that was the focus of this strategy (Cernusca & Mallik, 2018). However, the instructor observed a 
lack of consistency in students’ ability to analyze the conceptual structure behind the problem 
information and decide on the nature of the analysis that needs to be performed. To address this 
gap, during spring 2019 semester the instructor, working with the instructional designer, started 
to integrate in the instructional process a series of instructor-designed concepts maps associated 
with the worked examples prior to and part of the productive failure strategy. 

Concept mapping is an effective mindtool (Jonassen, 2000) that helps engage learners in 
critical thinking and deep learning (Bilik et al., 2020; Hay, 2007) and effectively assess learning 
(Weinerth et al., 2014). While concept mapping was introduced at a larger scale in education 
during 70’s, more current reviews of the literature show the potential impact of this tool across 
various areas of health professions such as medicine (Daley & Torre, 2010), nursing (Daley et 
al., 2016) and pharmacy (Hill, 2004; Noble et al., 2011). Finally, concept mapping proved to be 
an effective tool when integrated with other deep-learning instructional tool and strategies. For 
example, concept mapping was integrated with the use of a photograph association technique 
(Byme & Grace, 2010). While this integration was used to elicit children’s ideas about a specific 
science topic, concept mapping can be useful in pharmacy educational research associated with 
sensitive topics. For example, adding concept mapping to the use of photovoice strategy 
(Werremeyer et al., 2017) can enhance the quality of the outcome resulted from the photovoice 
intervention. Ata deeper level, Addae et al., (2012) proposed a modified Problem Based 
Learning (PBL) strategy where concept mapping was combined with the traditional PBL phases 
to produce a new 5-phase leaning approach in which three of the phases were designed as 
identifiable concept mapping tasks. In this strategy the first concept map was developed as a 
group map focused on the structure of the clinical problem and served as the center of the overall 
map that was developed in the other two concept mapping tasks. 

We decided to integrated concept mapping with the active learning in-class activities with 
a focus on increasing students’ ability to solve problems that were scaffolded with a productive 
failure strategy previously implemented in this course (Cernusca & Mallik, 2018). While the 
fully student-generated and partially student-generated concept maps proved to be more effective 
than the expert-generated concept maps (e.g. Lim et al. 2009) we decided to start this integration 
process with later type, expert-generated type with the instructor being the expert in this case. 
From an instructional impact perspective, the inclusion of instructor-designed maps was selected 
because it is a low-impact strategy that has a main goal to introduce students to the basic 
structure of the mental models associated with various stages in the problem-solving process. 
That is, this strategy was not expected to significantly increase the anxiety associated with the 
changes in the instructional process, reducing therefore the chances to create student resistance to 
the change. 

Examples of topics that were augmented with instructor’s concept maps were the 
identification of functional groups, molecule type identification or compound identification (see 
Figures 1 to 3). 
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Figure 1 
State of Matter concept map 
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Physical Properties of Molecules concept map 
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Figure 3 
Acid Base concept map 
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Because the structure of the instructor-generated concept maps was dependent of the 
context of the problem to be solved, the instructor encouraged students to try to adapt his maps 
for the problem that was solved during the classroom activities or as part of the assigned 
homework. 

Research Focus and Methodology 


Due to the relatively low impact of the use of instructor-driven concept maps, the major 
goal of this study was to explore the perceived impact of the integration of productive failure and 
concept mapping on pharmacy students’ self-efficacy (used as a proxy for students’ future 
performance). 


Research Design 
An exploratory quantitative design research was used to analyze if perceived impact of 
concept mapping and productive failure were significant predictors of student perceived self- 


efficacy. The proposed structural model is presented in Figure 4. 


Figure 4 
Proposed conceptual model related to the implemented instructional intervention 
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As shown in Figure 1, the expectation is that both the perceived impact of concept 
mapping and productive failure tasks on own learning increase student self-efficacy beliefs while 
the two types of strategies perceptions will interact with each other. 


Participants 

The course used for this study had an enrollment of 84 students in their first year of the 
Doctor of Pharmacy program. A convenience sampling strategy was used, all students in the 
course being invited to participate in the study. A number of 56 (67%) students volunteered to 
participate in this study. 
Data Collection 

The instructor collaborated with an instructional designer to administer, using 


Qualtrics®, an online an end-of-course survey with items adapted for three constructs validated 
in the literature. 
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The perceived impact of productive failure on own learning and the perceived impact of 
concept maps on own learning were implemented from Grasman & Cernusca (2015) with minor 
changes related to the course name and focal topic to customize them for this study (Appendix 
1). The self-efficacy construct was adapted from Cernusca & Price (2013). All three constructs 
used a 9-point Likert evaluation scale ranging from | for Strongly Disagree to 9 for Strongly 
Agree. Scores for each construct were computes as the average of the scores of its individual 
questions, resulting in a continuous score ranging from 1 to 9. The online survey was 
administered during the last two weeks of the course and students had 10 days to complete the 
survey. No bonus points were given for those that participated in this research study. This study 
was approved by the local Institutional Review Board and the informed consent form was posted 
at the onset of the survey indicating the voluntary participation in the study and the alternative 
task available for those interested in earning the bonus points but not participate in this study. 


Data Analysis 


Data collected were analyzed for basic statistics and correlations among proposed 
variables using SPSS v25. Analysis of raw data did not reveal outliers and the analysis of z- 
scores (< +/-2.5), skewness and kurtosis (< +/- 0.5) indicated an accepted level for the normality 
of the dataset. All three constructs adapted from the literature showed a very strong internal 
reliability with Cronbach’s Alpha values of 0.99 for perceived impact of productive failure on 
own learning and 0.95 for self-efficacy and impact of concept mapping on own learning. A path 
analysis model for the three proposed variables, perceived impact of productive failure on own 
learning, perceived impact of concept mapping on own learning and self-efficacy was tested 
using IBM SPSS Amos v25 software. 


Findings 


Table 1 presents the basic statistics for each of the measured continuous variable at the 
exit point in the course. As shown in Table 1, the proposed conceptual model was supported by 
the statistically significant (p<0.001) correlations among these variables with a moderate to high 
strength ranging from 0.58 to 0.64. 


Table 1 
Means, Standard Deviations, and Pearson Correlations for continuous variables (N=53) 
M SD 1 2 3 
1. Perceived impact of concept maps 7.24 169 - Oe" -264e* 
2. Perceived impact of productive failure 6.98 1.79 - o9F* 
3. Self-efficacy 7.04 1.44 - 


Note: **p < 0.01 (2-tailed) 


The results of the path analysis as resulted from the data generated with AMOS are summarized 
in Figure 5. 
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Figure 5 
Results for the proposed path analysis model 
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The path analysis indicated that both perception variables were significant predictors of 
self-efficacy, with perceived impact of productive failure standardized coefficient of B=0.32, 
p<0.01 and perceived impact of concept maps standardized coefficient of B=0.47, p<0.001. The 
covariance between the perception variables had a standardized coefficient of B=0.58, p<0.001. 
The overall model had a good fit (NFI=0.99; CFI=0.99) and the two perception variables 
explained 49% of the variance in students’ self-efficacy, an acceptable level considering the low- 
impact nature of the concept mapping instructional intervention. 

Because the exams in this course are open-notes, anecdotal data based on instructor’s 
observation during the exams indicated that students used the instructor-developed concept maps 
when they worked on the problems during the exam. 


Discussions and Further Research 


The results of this study indicate that the combination of the two strategies, productive 
failure and instructor-designed concept maps has the potential to significantly impact students’ 
perceptions and beliefs on own ability to perform well in the course. The acceptance of the two 
strategies is also a potential indicator of students’ willingness to move toward the use of deep 
learning problem-solving strategies as part of this course. 

Considering the piloting nature of this study, the research team intends to further explore 
the identified trends by expanding the use of instructor-driven concept maps and introduction of 
instructional tasks that will require students to generate their own concept maps for transfer 
problems associated with the worked examples integrated in the productive failure activities. 
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Appendix 1 
Instructional engagement constructs 
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Abstract 


To create a culture of deep learning needed to prepare pharmacy students for their roles 
involving patient care and team-based clinical decision-making concept mapping strategies were 
integrated in an entry-level and advanced-level pharmacy doctorate courses. Findings indicated 
that concept mapping strategies can help students build and represent complex conceptual 
models and that this strategy can be successfully adapted to match the level of skills and 
knowledge. Concept mapping also revealed areas where students need improvement in the 
organization and depiction of their knowledge. 


Background and Objective of the Study 


Pharmacists are continuosly expanding their role in health professions beyond the 
traditional roles of dispensing medications and medication management for patients. They are 
increasingly expected to be able analyze new information, draw valid conclusions and use them 
to solve complex problems in individual and population-level patient care. To prepare pharmacy 
students for these new professional roles, instructors need to integrate in the instructional process 
strategies that promote conceptual integration skills across various topics covered in their 
courses. In addition, current pharmacy accreditation standards emphasize problem-solving and 
creative thinking competencies which aligned with the new professional roles of pharmacists 
(Medina et al., 2013). 

Concept maps and concept mapping strategies are mindtools that have been shown to be 
effective in engaging students in the process of building complex mental models, similar to what 
is required in real-world problem solving situations and generating creative solutions to known 
challenges (e.g. Balaid et al., 2016; Bilik et al., 2020; Jonassen, 2000). While concept mapping 
was introduced at a larger scale in education during the 1970’s, more current reviews of the 
literature show the potential impact of this tool across various areas of health professions such as 
medicine (Daley & Torre, 2010), nursing (Daley et al., 2016; Garwood et al., 2018) and 
pharmacy (Carr-Lopez et al., 2004; Noble et al., 2011). 

Concept mapping proved to be an effective tool when integrated with other active 
learning instructional tool and strategies. For example, Addae et al., (2012) proposed a modified 
Problem Based Learning (PBL) strategy where concept mapping was combined with the 
traditional PBL phases to produce a new 5-phase learning approach in which three of the phases 
were designed as identifiable concept mapping tasks. In this strategy the first concept map was 
developed as a group map focused on the structure of the clinical problem and served as the 
center of the overall map that was developed in the other two concept mapping tasks. From an 
assessment perspective, concept maps proved to be very effective tools to measure both surface 
and deep-learning outcomes (Hay, 2007). 
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Finally, concept mapping proved to be an effective strategy not only for individual 
students but also for group instructional tasks (Mukherjee et al., 2018), as were implemented in 
the research being reported in this paper. 

Despite the many benefits of using concept mapping to increase depth of learning, 
concept mapping is a type of open-ended instructional strategy that can pose challenges to both 
instructors and students and therefore potentially create resistance to its use in the current 
pharmacy classroom. Lim and colleagues (2009) examined the effect of different concept 
mapping generativity level that ranged from expert-generated maps to partially learner-generated 
maps and to fully learner-generated maps. They found that while the students in the fully learner- 
generated concept maps outperformed the other two groups, those students with high self- 
regulating skills significantly outperformed those with low self-regulating skills. Therefore, 
staging the introduction of concepts maps starting with the lowest generative level, expert- 
generated maps and increasing the engagement with the next levels of generativity can help 
decrease the gap between various level of self-regulating skills specific for a student cohort, 
especially in large courses. The major objective of this study was to explore to what degree 
concept mapping strategies can be integrated in both entry-level and advanced pharmacy courses 
as part of a pharmacy doctorate curriculum. 


Instructional Design Interventions 


During the Fall 2018 semester, an instructional designer worked with two instructors to 
implement concept mapping strategies in two courses across a pharmacy doctorate (PharmD) 
curriculum. The first course was Public Health for Pharmacists, an advanced-level course for the 
third year (P3) PharmD students and the second was Pathophysiology, an entry-level clinical 
course for first year (P1) PharmD students. 

Because of the advanced level of the course and its structure with a lecture module and an 
active learning module, the concept mapping intervention in the P3 course was integrated in two 
stages. At a first stage, in the lecture part of the course the instructor developed a complex 
concept map that synthesized the main elements related to the role of pharmacists in ensuring 
population health, the focus of the course (Fig.1). However, due to its complexity the concept 
map was not presented in the beginning of the course in its totality but rather was integrated as 
sub-maps throughout the lectures. Figure 2 shows a partial map presented and discussed with 
students as the semester progressed. The goal of the partial concept map integration into the 
lecture was two-fold. First, the instructor introduced students to concept mapping strategies by 
providing example of maps ranging from simple to complex. Second, the instructor used the 
concept maps to both emphasize the interconnectivity and complexity of the course topic and to 
generate class-wide discussions about the course topics. These discussions had the additional 
outcome of making edits and ultimately, completion of the partial and the final concept maps. 

As a second step, in the active learning part of the class the instructor used one of the 
major team-based case studies addressing the analysis of the burden of Hepatitis C (Hep C) on 
the population health system as the context for a student-generated concept mapping task. 
Student teams were tasked with the development of a concept map that reflected strategies that 
pharmacists can use to help in reducing the Hep C burden and improve population health. The 
major goal of this activity was to engage students in course-related tasks that were beyond the 
traditional role of a pharmacist and consequently to increase students’ beliefs in the usefulness of 
the skills they built in this course. 
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The active learning part of the course was held in a technology-rich classroom that had a 
SCALE-UP design with round tables that host groups of up to 9 students. Each team used post-it 
notes provided by the instructor to write the major concepts on and had access to a whiteboard or 
a big empty post-it-note to build their concept map (see Figure 3). 


Figure 1 
Pharmacists’ role in ensuring population health. Instructor-developed concept map. 
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Figure 2 
Pharmacists’ role in ensuring population health. Partial concept map integrated in lectures. 
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As shown in Figure 3, each group found the best strategy to collaborate on integrating the 
knowledge built during the lecture and from the analysis of the Hep C case study into their own 
concept map. 


Figure 3 
Concept mapping activity. Working as groups to build HepC concept maps. 


At the end of the activity, one representative from each group presented their group’s 
concept map to the entire class (Fig.4). 


wa 


Figure 4 
Concept map presentation for the entire class 


Finally, the instructor showed his own 
concept map and used it to generate discussion 
about the topic that was the focus of the in-class 
concept mapping activity. Because of the 
relative novelty pf the concept mapping 
strategy, the resulted concept maps were not 
included in the course assessment strategy. 

For the entry-level P1 course, the 
opportunity to test the potential of the concept 
mapping activity occurred toward the end of the 
semester. The timing of this activity was 
considered optimal because most students were 
enrolled simultaneously in both the immunology 
and the pathophysiology courses. 
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Therefore, the topic of the lecture integrated a significant amount of instructional material 
that was already covered in the immunology course. Consequently, the implementation of 
concept mapping activity was less structured than in the previous course. To build their concept 
maps, students were required to focus on the major relationships between immunology and 
pathophysiology through the perspective of the role of the pharmacist by integrating these two 
areas of study. Students were able to actively refresh immunology concepts while attempting to 
link them with pathophysiology concepts. Compared to a standard lecture, students were 
expected to be a significantly more engaged in the topic as a result of overlap of topics and 
courses. During the first half of the class period, teams of students engaged in building the 
concept maps with a minimum introduction by the instructor of what a concept map is. White 
boards were used to provide a space for each team to collaborate in the development of their 
concept map. In the second half of the class period, the instructor built on the concept mapping 
activity to expand, emphasize, and reinforce the major relationships between immunology and 
pathophysiology concepts from a pharmacist’s practice perspective. The resulted concept maps 
were not included in the grading scheme of the course. 


Research Focus and Methodology 


Due to the relatively low impact of the use of concept mapping activities in the two 
courses, the major goal of this study was to evaluate the potential impact of the concept mapping 
on students’ ability to represent the dept of their learning at various levels across the curriculum. 


Participants 


A convenience sampling was used for this study. For the P1, entry-level course, the class 
size was 78 students and all of them volunteered to participate in the study. For the P3 course, 
the class size was 89 and 87 (98%) students volunteered to participate in the study. 


Data Collection and Analysis 


All group concept maps were captured by the instructor as a photograph at the end of the 
activity. There were 18 groups for the P1, entry-level course, and 11 groups in the P3, advanced- 
level course. Considering the exploratory nature of this intervention and the span of knowledge 
and skills of the two groups, to evaluate the overall quality of the concept maps, an evaluation 
rubric was developed with three assessment dimensions adapted from the literature (e.g. 
Jonassen, 2000): accuracy of instances (concept-link-concept) used in the concept map, depth of 
the map structure, and connectedness of the map structure. Each dimension was defined and 
scored ranging from | (low) to 5 (high) as shown in Table | below. To determine the final map 
score, the three scores were summed and computed as percentage of the total potential score. 

In addition, to check if the course-related activities could impact the overall development 
of the concept maps, students’ course self-efficacy was measured using items adapted from 
constructs validated in the literature. The self-efficacy construct was adapted from Cernusca & 
Price (2013). The self-efficacy construct used a 5-point Likert evaluation scale. Scores for each 
construct were computed as the average of the scores of its individual questions, resulting in a 
continuous score ranging from | to 5. 
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This construct was part of an exit online survey administered in Qualtrics® during the 
last week of the course. Self-efficacy data and cohort analysis were performed with SPSS v25®. 


Table 1 
Concept map scoring rubric 


Accuracy of instances (concept-link-concept) 
5 — at least half of the instances (concept-link-concept) have both the concepts and links 
labeled correctly as single term node or link significant for the topic at hand 
3 — one third or more of the instances (concept-link-concept) have both the concepts and 
links labeled correctly as single term node or link significant for the topic at hand 
1 — less than one third or more of the instances (concept-link-concept) have both the 
concepts and links labeled correctly as single term node or link significant for the topic at 
hand 
Depth — the level of hierarchical links 
5 — at least half of concepts are linked in 2 or more hierarchical levels (from general to 
specific) 
3 — one third to about half of the concepts are linked in 1 or 2 hierarchical levels 
1 — less than one third of the concepts are linked in two hierarchical levels 
Connectedness — ratio to dead-end to multiple-linked nodes 
5 -high ratio of multiple-linked to dead-end nodes indicating a good understanding of the 
complexity of the topic 
3 - dead-end nodes are same or slightly more than multiple-linked nodes indicating a fair 
understanding of the complexity of the topic 
1 —dead-end nodes are significantly more than multiple-linked nodes indicating poor 
understanding of the complexity of the topic 


Findings 


The qualitative evaluation of the student-team concept maps indicated that, relative to the 
complexity of the task for each course, the maps had a relatively high complexity and correctly 
represented the conceptual models associated with the focal topic. As expected, due to the 
differences in the academic level of the two groups and the complexity of the concept map 
integration in the instructional process the scores of the P3 cohort (67% to 93%) were overall 
higher than the one for the P1 group (27% to 87%). The main evaluation dimension that 
differentiated the two groups was the accuracy of the instances in the map structure. Most of the 
concept maps built by the P1 student groups had correctly labeled nodes with individual concepts 
but the links between the nodes were not labeled and often were not directional. However, two of 
the P1 groups scored 4 and 5 on the accuracy dimension. 

On the other hand, the connectedness of the maps for the Pl group was much closer to the 
P3 group with more than two thirds of the connectedness scores for P1 concept maps being at or 
above 3, the average of the evaluation score for this dimension. Figure 5 shows the concept map 
developed by the group that earned the highest score. As a comparison, Figure 6 shows a P1 map 
with a low score, while Figure 7 shows the highest scored concept map developed by a P3 group. 
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Figure 5 
PI concept map that had the higher score in their cohort 


Figure 6 
PI concept map that had one of the lowest scores in their cohort 
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Figure 7 
P3 concept map that had the higher score in their cohort 


These findings show that 
while clearly the quality of the 
concept maps are higher when the 
concept mapping strategy is 
integrated in the instructional 
process and the task is part of an 
active learning activity, as was the 
case with the P3 group, where 
students were able to represent 
with concept mapping the 
complexity of their knowledge 
even when this strategy was 
introduced as an ad-hoc task in 
the instructional process. 

For the self-efficacy data, 
the construct used had weak 
internal reliability for P1 cohort 
with Cronbach’s alpha of 0.42, 
but a very high internal reliability 
for the P3 course with Cronbach’s 
alpha of 0.93. Both cohorts 
reported above the mean to high 
self-efficacy levels, with P1 
students having an average score 
of 3.87 (SD=.46) and P3 students 
an average score of 4.03 
(SD=.62), on a | — low to 5 — high self-efficacy evaluation scale. An independent sample t-test 
showed no statistically significant difference was found between the mean self-efficacy scores of 
the two cohorts, t(163)=-.189, p = .06. Also, a one-sample t-test indicated that both cohorts 
reported self-efficacy scores statistically significantly higher (p < .001) than the mean (3) of the 
evaluation scale. This suggests that both courses provided students with similar engagement and 
feedback opportunity to increase their beliefs in the ability to learn in their course. 


Conclusion 


The finding of this exploratory study indicated that concept mapping strategies can help 
students build and represent complex conceptual models relating complex roles of pharmacists 
that go beyond the traditional medication dispensing roles. This strategy can be successfully 
adapted to match the level of skills and knowledge for both entry and advanced-level pharmacy 
students. Future research will focus on the concept mapping to provide instructors with a clearer 
picture of areas that need improvement to further help pharmacy students prepare for their 
professional journey, especially for advanced-level courses. 
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Abstract 
Designing experiential learning activities requires an instructor to think about learning outcomes. 


Using importance-performance analysis can assist with the instructional design of the activities. 
This study used IPA in a fully online asynchronous graduate class. The primary goal of this 
article is to demonstrate the use of IPA to guide the instructional design process of e-service 
learning activities. 


Keywords: e-service learning, online course design, experiential learning, importance- 
performance analysis 


Introduction 
Service-learning has been a proven learning strategy from the days of Dewy and is often 


intertwined with other forms of experiential learning such as practicums and work-related 
learning. Caspersz and Olaru (2015) define service-learning as a “process of reflective education 
in which students learn civic or social responsibility through a scholarship of community 
engagement that embodies the principle of reciprocity” (p. 19). In practice, service-learning 
integrates academic rigor with real-world projects that students undertake with community 
groups and thus creates a reciprocal learning relationship between the students and their 
community groups. Typically, community-engaged pedagogies such as service-learning require 
facilitative expertise that educators develop and refine through scholarly teaching that integrates 
experience and reflective teaching (Strait, Turk, & Nordyke, 2015; Waldner, 2015). 

Service-learning is often integrated into face-to-face environments but with the number 
of online courses and programs, creating high-impact practices for online service-learning is 
imperative. E-service learning occurs when the instructional, service, or both components are 
conducted online. For example, students in an online history class may assist a museum with 
digital archives. Dailey-Herbt and colleagues (2008) described e-service learning as a pedagogy 
that engages learners through technology. Previously, it was not easy to engage the community 
from a distance, but technology development, connecting students from a distance with 
community-based practices, is seamless. 
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Designing for Online Service Learning 


With the increase of online education, many studies focus on transitioning from 
traditional service-learning to e-service learning. Often educators find the online environment as 
a barrier for effectively implementing service-learning. Frameworks have been proposed for 
implementing e-service learning, yet there is no "one size fits all" due to the diverse needs of 
projects. Researchers hope that the design and development of e-service learning will help future 
educators make informed decisions and plan the best implementation for their respective 
projects. E-service learning challenges educators to make full use of technology rather than 
replacing instructors with technology. The focus should be on the design of the course and 
project. 

Although there is a plethora of research regarding service-learning benefits, research on 
online service-learning is lacking. Marcus and colleagues (2020) conducted a systematic review 
of e-service learning literature and found only eleven studies that focused on the design and 
development of service-learning for online courses. Many of the studies offer suggestions for 
designing e-service learning for various disciplines. Strait and Sauer (2004) offered suggestions 
for beginners such as starting small with one course, introducing learners to the purpose, 
providing explicit communication channels for both the learners and community partners and 
ensuring flexibility. Bangert (2004) focused on the reflective portion of service-learning, stating 
discussions should be carefully crafted to create stimulating discourse and groups monitored to 
ensure equal participation. Helms and colleagues (2015) also focused on group communication. 
They recommended creating a group charter for students to develop a plan for working together, 
including developing a communication plan. Students may have to identify community partners 
or clients since many students may live outside the school area they are attending; it was also 
suggested to provide a list of criteria the project must meet. Students should submit a proposal 
early in the semester with the organization and explain why this organization meets the criteria. 

Overall, the best practices suggested by researchers are also best practices for online 
course design and development, as suggested by Quality Matters (QM) ™. QM ™ is a faculty- 
oriented, process-centered, peer review process developed from research based on instructional 
design principles to assure quality design in online and blended courses. Many of the suggestions 
offered for designing e-service learning courses are incorporated within QM rubric's best 
practices. For example, introducing the learners to the purpose is covered in specific standard 
1.2. Creating engaging and meaningful discussions is covered in specific standard 5.1. Specific 
standard 5.4 addresses communication among learners. Although researchers have offered some 
specific examples, such as the use of a group charter (Helms et al., 2015), designing a course to 
meet QM standards, can significantly assist an instructor in any discipline with designing and 
developing a course to support e-service learning. 

This study analyzed student perceptions of an e-service learning project to inform future 
course design. 
Methods 

This quasi-experimental study explored students’ perceptions regarding online 
experiential learning in terms of what they perceived to be important and how they performed. 
This study utilized an importance-performance analysis (IPA) method. Additionally, students 
were required to complete a reflection at the end of the semester. The reflections were analyzed 
to garner additional information about the students’ experience with service learning in an online 


asynchronous class. 
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Participants 

This study used a graduate course in an Instructional Technology program delivered in 
Spring 2020 during a fifteen-week semester. The online asynchronous course had 14 students. 
Students were recruited to participate in this study voluntarily at the beginning of the semester. 
Students who chose to participate in the study were assigned a random identification number, 
which allowed the researcher to match survey data anonymously. Twelve students completed 
both the importance and performance survey, with an 85% response rate. This response rate falls 
in line with the recommended response rate for online surveys (Baruch & Holtom, 2008). 


Importance-Performance SELEB Survey 

Importance Performance analysis has been used primarily in marketing and tourism yet 
has found a place in higher education for student evaluation of teaching and course design 
(Anderson, Hsu, & Kinney, 2016; Caspersz & Olaru, 2015; Huybers, 2014). Alberty and Mihalik 
(1989) also stated that Importance-Performance analysis has an important place in adult 
education. Important-Performance surveys allow students to evaluate both constructs of various 
teaching attributes (see Figure 1). Specifically, the importance-performance model identifies the 
relative importance of factors influencing student preference (importance component) and 
indicates the degree to which a particular instructor or setting possesses these factors 
(performance component). 
Figure 1 
Importance-performance analysis (IPA) 
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The SELEB (Service-Learning Benefits) survey developed by Toncar and colleagues 
(2003) was used to gauge student perceptions of the service-learning with minor word changes to 
terminology (Caspersz & Olaru, 2017). The SELEB scale consists of fifteen items that fall under 
four main factors: practical skills, interpersonal skills, citizenship, and personal responsibility. 
The Importance SELEB scale was given to the students at the beginning of the semester. The 
scale is 1 — 5, with one being extremely important and five being not at all important. The 
Performance SELEB was deployed at the end of the semester upon completion of the 
experiential learning activity. Table 1 displays the survey items and how each is associated with 
each main factor. There are four factors associated with the fifteen variables. 


Table 1 


Factors and Associated Variables of the SELEB Scale 


Practical Skills Interpersonal Skills Citizenship Personal 
Responsibility 
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Apply information 
learned in the classroom 
and apply it to real-life 
scenarios 


Apply problem-solving 
techniques 


Build my self-confidence 


Learn practical workplace 
skills 


Develop organizational 
skills 


Experience 
personal growth 


Enhance my 
leadership skills 


Further develop 
my oral and 
written 
communication 
skills 
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Gain a greater 
understanding of 
cultural and racial 
differences 


Develop social 
responsibility and 
citizenship skills 


Be involved with the 
community 


Make a difference in 
the community 


View social 
issues from a 
variety of 
perspectives 


Demonstrate my 
trustworthiness 
to others 


View social 
issues from a 
variety of 
perspectives 


Service-Learning Project Description 


The project was implemented in a fully online asynchronous graduate instructional 
technology course on change management. The course was designed to align with the QM 
Rubric and was informally reviewed by a certified QM Peer Reviewer. Throughout the semester, 
students were asked to reflect on their experience every three weeks with a cumulative reflection 
at the end of the course. The students were also expected to complete a service-learning project 
for a local non-profit organization. 

The class teamed up with a local non-profit to assist with implementing a technology 
change within the organization. The instructor chose the community partner, and students formed 
teams to tackle the problem collaboratively. Students were required to attend a kickoff meeting 
(synchronously) with the administrative staff from the non-profit organization to review the 
organizational needs. Since the entire class was working with one client, it was determined that 
all communication, including questions, would go through the instructor, and questions would be 
sent to the client once a week on Thursdays with an expected response time on the following 
Tuesday. Throughout the semester, the students were required to submit various checkpoints for 
their final Change Management Plan. 

Results 

IPA was used to analyze student perceptions and Performance. This data was used to 
construct a two-dimensional matrix. Importance is depicted on the y-axis, and Performance is 
depicted on the x-axis. The IPA graphs' gridlines are determined by the overall mean of 
Importance (4.40) and Performance (4.23). 

A paired-samples t-test was conducted to evaluate the impact of the experiential learning 
assignment using the importance-performance scale. There was no significant difference 
between importance and Performance for all associated variables. Table 2 displays the mean 
scores for the four main factors of the SELEB survey. There was a decrease in the mean from 
importance to Performance in interpersonal skills and citizenship and an increase in personal 
responsibility. 


Table 2 
Mean comparison of the SELEB factors 


Factors Importance Performance 
Practical Skills 4.58 4.58 
Interpersonal Skills 4.52 4.33 
Citizenship 3.83 3.81 
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Personal Responsibility 3.89 4.22 


A paired-samples t-test was conducted to analyze the four main factors comparing the 
importance to the performance. Practical skills remained the same from importance (M/ = 4.58, 
SD = .452) to performance (M = 4.58, SD = .452), t (9) = .000, p = .1.000 (two-tailed). The mean 
decrease in the practical skills scores was -1.293 with a 95% confidence interval ranging from 
458 to 2.112. The eta squared statistic (0) indicated a small effect size. Interpersonal skills 
resulted in a decrease from importance (M = 4.52, SD = .444) to performance (M = 4.33, SD = 
577), t (9) = 1.644, p = .139 (two-tailed). The eta squared statistic (.37) indicated a moderate 
effect size. Citizenship skills resulted in a minor decrease from importance (M = 3.83, SD = 
.750) to performance (M = 3.80, SD = .933), t (9) = .096, p = .926 (two-tailed). The eta squared 
statistic (.035) indicated a small effect size. Personal responsibility skills resulted in an increase 
from importance (IM = 3.89, SD = .943) to performance (M = 4.22, SD = .687), t (9) = -1.664, p = 
.135 (two-tailed). The eta squared statistic (.40) indicated a small effect size. (see Table 3) 


Table 3 

Paired t-test important-performance of SELEB factors 
Factor t Sig. (2-tailed) Eta Squared 
Practical Skills 0.00 1.00 0 
Interpersonal Skills 1.64 139 Si 
Citizenship .096 .926 .035 
Personal -1.66 mt ee 40 
Responsibility 


Table 4 depicts the analysis of the two-tailed t-test for the variables of the four main 
factors. Each item is labeled which is depicted in Figure 2 on the IPA quadrant. There is a 
decrease in the mean scores from importance to performance. 


Table 4 
t-test importance-performance of SELEB items. Letters correspond with F igure 2 


SELEB Items Mean Mean t Sig. (2- Eta 
Importance Performance tailed) squared 
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Apply information 
learned in the 


classroom and apply it 


to real-life scenarios 


Learn practical 
workplace skills 


Develop organizational 


skills 


Gain a greater 
understanding of 
cultural and racial 
difference 


Develop social 
responsibility and 
citizenship skills 


Experience personal 
growth 


Be involved with the 
community 


4.67 


4.67 


4.44 


4.11 


4.00 


4.67 


3.44 


4.67 0.00 
4.67 0.00 
4.56 -1.00 
4.22 -.426 
3.67 -.426 
4.44 .800 
3.67 -.686 
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1.00 


1.00 


347 


.681 


.681 


A447 


po Be 
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*p < .05, two-tailed [LP1] 


Figure 2 illustrates the associated variables from the importance and performance SELEB 
scale on an importance-performance quadrant. The x and y axis are determined by determining 
the overall mean score for both importance and performance (Huybers, 2014). The overall 
importance mean was 4.40 and overall performance was 4.23. Applying real world skills (A), 
practical workplace skills (B), organizational skills (C), personal growth (F), self-confidence (K), 
and problem solving techniques (N) landed in the upper right quadrant, which suggests that this 
activity met both the importance and performance for students. Leadership (L) and development 
of oral and written skills (M) fell in the lower right (concentrate here) quadrant. Greater 
understanding of culture and racial differences (D), social responsibility (E), community 
involvement (G), make a difference in the community (H), view social issues from a variety of 
perspectives (J), and demonstrate trustworthiness to others (O) fell in the lower left (low priority) 
quadrant. 


Figure 2. 
Results from the importance-performance means 
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Discussion 
There are many reasons for implementing service-learning to enhance student learning 


outcomes and various other skills, such as the development of social responsibility. Within the 
online environment, service-learning can prove challenging. Incorporating Importance- 
Performance analysis can assist the course designer/instructor with design suggestions focusing 
on specific skill development. For example, leadership and oral and written communication skills 
were an area that needs concentration. The course designer can take this information to enhance 
student learning in these areas. Each student was in a team for this course; therefore, many of the 
students should indirectly be working on both leadership skills within the group and the client 
and their oral and written communication. The course designer will revisit these areas to develop 
more concrete examples of the expectations of working with a team of peers and how to interact 
with a client to ensure the students stretch their knowledge and improve already existing skills. 
With the areas that fell into the ‘low priority’ quadrant, the course designer/instructor needs to 
analyze the skills and determine if they are pertinent to the course/program goals. An instructor 
can focus on specific areas for student development through IPA and build those specific 
concepts into the course design. 


Limitations 

There were a few limitations with this initial study. First, it was a small class but as Strait 
and Sauer (2004) stated, it is important to start small then apply appropriate design to new 
courses. Next, during the middle of the semester the COVID-19 pandemic put additional stress 
on many of the students either due to adding homeschooling to their daily activities along with 
graduate schoolwork and full-time employment. During the last half of the semester, many 
students focused their reflections on venting about COVID-19 and the undue stress that came 
with it. Some of the COVID-19 stress/fatigue affected group dynamics. Finally, the class was 
given a project to work on rather than choose their own project. This could have influenced the 
degree the students felt invested in the project and with the client. 
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Abstract 
We investigated how: (a) anonymous versus identified online discussions, in elementary 
school, compared for task relevance of postings, and (b) how game-like conversations 
with story characters affected task relevance of postings. We conducted three similar 
studies involving fourth graders reading short web-based eBooks with small group 
online discussions and game-like conversations with characters. All discussion postings 
were qualitatively coded for discussion relevance. Students in name-identified 
discussions were significantly more on task, than those in anonymous discussions. 
Students who could participate in game-like conversations with story characters were 
also significantly more on task, than those who did not. In times of social isolation, 
when school is increasingly conducted remotely online, the effective design of small 
group online discussion is vital. 


Key words: online discussion, elementary school, online education, distance education 


Introduction 


Reading is on the decline in the Western world, including in USA and Slovenia (OECD, 
2010; Mol and Bus, 2011). Children’s literacy skills in the USA have stagnated from 
2000 to 2020 (PISA, 2019). Children’s recreational reading, the most important factor 
in developing literacy skills, is on the decline (Mol & Bus, 2011). Meanwhile, the 
popularity of computer games and social media has soared among children. How can we 
reinvent the book technology to tap into children’s digital recreational interests, to 
increase motivation to read and literary skills? Computer games provide a mindset that 
allows children be more persistent with higher level cognitive challenges (Gee, 2005). 
Therefore, incorporating games and game-like interaction into web-based eBooks has 
potential to motivate children to read. Reading is often individual, one person curled up 
on a couch with a book, escaping into imagined worlds (Zwaan, 1999). However, 
reading is also social, for instance conversations over coffee about books. For the 
younger generation, much conversation is digital, SMS (Short Message Service) or 
social media. To stimulate children’s recreational reading, we invented a new form of 
book, adding games and social interaction into web-based eBooks. Students read pages 
of text, come to a game that can only be won by comprehending the previous text, then 
read on. Recently, we added social interactions to these eBooks, via small group texting 
discussions. 
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Research questions 


We focus on these questions about interactive eBooks: (a) how do different forms of 
interaction (individual and social) influence each other, i.e., “How does individual 
game-like interaction influence social small-group discussions?” and (b) “How do 
staging factors influence the quality of small group online discussions?” 


Method 


The first author received a Fulbright Scholarship to conduct research in Slovenia on 
web-based eBooks. In 2019, he conducted two studies in Slovenia, where students read 
and interacted with very short eBooks based on stories from the fourth-grade textbooks 
used in schools. In each classroom, the intervention lasted two class periods in a single 
session. First, we conducted two pilot studies (total of four classrooms), in Slovenia, 
investigating how fourth grade students, reading web-based eBook stories, interacted 
individually and socially within the eBooks. In each of three stories, students (a) 
engaged in game-like conversations with characters, and (b) answered open-ended 
questions, and (c) interacted socially with each other in small group online discussions 
within eBooks. Each eBook had both individual and social elements. Students took a 
pre- and post-test on reading motivation. 


Secondly, we conducted a full study in 19 Slovenian classes. The full study included 
three treatment groups: (a) individual interaction only (game-like conversations with 
characters and open-ended questions), versus (b) social interaction only, versus (c) text- 
only. In the full study, students also received real-time feedback to open-ended 
questions using Natural Language Processing (NLP). We developed NLP algorithms 
first in English, using questions and answers in English. We then successfully 
transferred the NLP algorithms to Slovenian, to provide feedback to students answers to 
Slovenian questions. 


Results 

We report the quantitative results, briefly at the conference. However, we focus 
on qualitative results: (a) coding of the small-group discussions, (b) how effectively the 
NLP was used to provide feedback to open-ended questions. 


Small-group discussions: More than 3,000 postings from small group discussions were 
qualitatively coded along three dimensions: (a) book relevance — whether the posting 
related to the story, (b) type - whether the posting is written is a question, an answer or a 
statement, and (c) category - a very simplified “descriptive” of what the text was about. 


There were dramatic differences between the first pilot study and the full study. In the 
first pilot study, 68.8% of discussion postings were coded as book relevant; 31.2% were 
NOT book relevant. By contrast in the full study, 34.5% of discussion postings were 
book relevant, 65.5% were NOT book relevant. Similar dramatic differences were found 
between pilot one and full study in type and category. 


Discussion 
There are two differences between pilot study one and the full study, and thus two 


possible causes for dramatic differences: (1) in the pilot studies, each student 
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experienced all forms of interaction, both individual and social, within all three stories: 
game-like conversations with characters, answered open-ended questions, and small 
group online discussions, and (2) in pilot study one, the student usernames for logging 
in to the eBook system were their first and last names; while in the full study, because 
IRB requirements, usernames were anonymous. 

Two possible causes in more detail: 

(1) Classroom observations in pilot study one suggest students were excited by 
the content-related challenges of game-like conversations with characters and open- 
ended questions. This excitement carried over into the small group online discussions 
where students discussed in lively manner. In the full study, there were three different 
treatment conditions: (a) individual interaction only (game-like conversations with 
characters and open-ended questions), versus (b) social/online discussions only, versus 
(c) text-only. Thus, students in the discussion condition, read the almost the entire story 
in text-only form and, towards the end of the story, engaged in the online small group 
discussions. Classroom observation suggest there was no excitement while reading the 
story in text-only form, and thus no challenge/excitement that carried over into small- 
group online discussions. 

(2) In the full study, because of anonymous usernames, students either: (a) first 
spent time, asking who others were, before discussing the story, or (b) assumed that 
anonymity freed them from texting in a socially acceptable manner. In these cases, the 
moderating influence of the researchers or teachers, sometimes brought the students 
back on track. 

In pilot study two, students had anonymous user names, but each student 
experienced all forms of interaction, both individual and social, within each of the three 
stories. Thus, the coded discussion postings from pilot study two may shed light on 
which of the two causes is more probable. As of this writing, that analysis has not yet 
been done. However, these, and other results, will be revealed at the presentation. 
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Abstract 

Human beings' desire to learn and improve their life standards without any time and place 
dependence has resulted in the spread of MOOC applications. MOOCs have become one of the 
mainstream lifelong learning opportunities for students, adults and anyone who wish to improve 
herself. Increasing interest in MOOCs necessitates examining of their quality. So, this study 
intended to investigate the learners’ experiences in MOOCs and the future course 
recommendations. The learners’ experiences in MOOCs was grouped under three categories: 
content and design, course site and media, and instructor behavior. Some of the most satisfying 
experiences of the learners were variety of the course topics, interactive contents, easy access to 
the content and flexibility, personalized feedback from the real instructors and their expertice in 
the content areas. Meanwhile they like to see more courses on technology related topics, such as 
coding, information technologies, and some Adobe products. 


Introduction 


Advances in information technologies have effected the way we teach and learn, and 
consequently new learning environments and implementations have emerged. Massive Open 
Online Courses (MOOCs) is one those emerged learning environments. MOOCs, which can be 
considered as an implementation of connectivist learning theory (Siemens, 2013) and have a wide 
repercussion around the world, offer learners from all over the world the opportunity to learn 
through free, online and open courses (Kaplan & Haenlein, 2016). Today, we see that MOOC 
applications are becoming widespread in order to put more information into circulation and to make 
the courses at universities accessible and flexible for more people. Latest figures reveal that 
currently there are more than 8000 MOOC offerings from more than 800 institutions to around 60 
million users globally. As a reflection of this global movement, several MOOCs platforms have 
launched in Turkey. For instance, AKADEMA, a MOOCs platform was launched by Anadolu 
University in June 2014 (Mutlu at al., 2014). Based-on long history of the open and distance 
learning experience, Anadolu has been designing and offering a number of online courses to 
anybody who wish to learn without any prerequisites and fees. The courses, unlike many other 
MOOC platforms, are facilitated by the real university instructors and the learners need to interact 
with them as well as other learners to be able to complete the course activities. So, a big majority 
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of the courses employed a guided-study approach except couple self-paced ones. Currently, 121 
courses are being offered in AKADEMA, almost all are in Turkish expect two new ones in English. 
Anadolu University considers AKADEMA as a social responsibility project to share its knowledge 
to the society. 

MOOCs are getting more interest due to the fact that traditional education systems are 
falling short on meeting the needs of the learners and institutions (Koutropoulos et al., 2012). This 
increasing interest of learners has also grasped the researchers’ attention. Since the beginning of 
2010, we have seen more research studies examining the effects of MOOCs on learning processes 
(Gasevic, et al., 2014; Sa’don, Alias, & Ohshime, 2014; Sangra, et al., 2015; Loizzo & Ertmer, 
2016). In those studies, self-regulation, social learning, learner-to-learner interaction, learner 
motivation, completion, learner characteristics, cultural differences, diversity, learning in social 
networks, MOOC design, learning design, and similar topics are the ones often investigated. 
However, as it was mentioned in the literature in order to understand a phenomenon, we need to 
examine it under different settings and circumstances. Therefore, this study proposes to learn more 
from the actual learners’ experiences in MOOCs offered in AKADEMA platform to be able to 
improve the learning processes in these courses. 


Purpose and Method 


This study intended to investigate the learners’ experiences in MOOCs offered in the 
AKADEMA platform of Anadolu University and their future course recommendations, or the 
topics they are interested in learning through MOOCs. So, the research questions were formulated 
as following: 

1. What are the components and characteristics of MOOCs in AKADEMA that the learners 
satisfied the most? 
2. What kinds of topics would the participants like to learn in AKADEMA MOOCs? 


A descriptive content analysis approach was conducted in the study. Content analysis is to 
gather similar data within the framework of certain concepts and themes and to interpret them in a 
way that the reader can understand (Yildirm & Simsek, 2006). In other words, content analysis 
can be defined as a systematic, repeatable technique in which some words of a text such as book, 
book chapter, article, thesis, letter, historical documents, newspaper titles and articles are 
summarized with smaller content categories with coding based on certain rules. Content analysis 
can be used in both qualitative and quantitative research. The steps followed in the research process 
and the way the data are collected reveal whether the study is suitable for quantitative or qualitative 
method. In this study, the quantitative method was used because a series of descriptive statistics 
was used during the analysis. 

An online survey with two major open-ended questions was used to collect data. One of the 
questions asked the learners indicate their perceptions regarding the most and the least satisfactory 
characteristics of the MOOCs. The second question allowed the participants to state the topics they 
are interested in learning. The data collected during May 2018 with voluntarily participation of 
learners in AKADEMA MOOCs. Total 719 participants responded these two questions and 
included into the analysis. After data collection, the data was transformed into a script and then 
two coders separately analyzed. Two coders, first, created a code schema and then with the 
participation of a third expert (one of the authors of this paper) compared and synthetized. Then 
themes were derived from the codes (Figure 1). 
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Figure |. Data analysis process 
Findings 


The reporting of the findings was organized according to the research questions. Figure 2, 
first of all, was related to the question about to what extent the courses met the learners’ 
expectations. As can be observed from the figure, quite a number of participants (66 percent) 
indicated that their expectations were met in the AKADEMA courses while still one fifth of them 
(20 percent) were not happy about the courses. A higher percent of the learners also stated that they 
did not have any problem or experience any issue during the learning processes. 


Partly Yes 
by0) 


Figure 2. To what extent the courses met the participants’ expectations 


Table 1, on the other hand, summarizes the most satisfied components or characteristics of 
the courses the participants took and completed. 


Table 1. The most satisfied components and characteristics of MOOCs in AKADEMA 


Theme F 

1. Content and Design 

Variety and up-to-dateness of the contents 52 

Interactivity in the contents 38 

Appealing of the contents 25 

Fruitfulness of the content for learning eal 

Fun in courses 22 

Facilitation of the personalized development 12 
2. Course Site and Media 

Accessibility 32 

Flexibility to reach the course site and resources 26 
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Interactive videos 22 


Group interaction 20 
Sense of classroom community 12 
Providing discussion opportunities 10 
3. Instructor Behaviors 

Personalized feedback 18 
Content expertise 12 
Accessibility 10 
Teaching presence 2 


As can be observed in Table 1, the participants’ responses to this question was classified 
under three major themes: content and design, course site and media, instructor behaviors. 
According to the results, a majority of the participants satisfied with the variety and up-to-dateness 
of the content in the courses (f=52). Interactive design of the courses (f=38) and attention-grabbing 
characteristic of the content (f=31) were also other satisfying features of the MOOCs. In terms of 
course site and instructional media or materials provided to the learners, the most satisfying 
characteristics of the courses listed as accessibility (f=32), flexibility to reach the course site and 
resources (f=26), interactive videos provided (f=22), promoting small group interaction (f=20), and 
sense of classroom community (f=12), and finally discussion opportunities provided throughout 
the course (f=10). The final theme was about the instructors’ actions or behaviors during the 
courses. Under this theme, four characteristics stepped forward in the learners’ answers. A number 
of learners (f=18) indicated their satisfaction with the instructors’ personalized feedback for their 
learning progresses. Similarly, the learners also pointed out their satisfaction with the instructors’ 
expertise in their fields (f=12). They also felt high level of sense of teaching presence in the courses 
and satisfied with this feeling. 

The following table (Table 2) presents the topics and courses the participants would like to 
learn or take as MOOCs in the AKADEMA platform. 


Table 2. Topics and courses the participants would like to learn in AKADEMA 


Theme F 

1. Technology 

Adobe Products 22 

Coding 15 

Introduction to Computing 12 
2. Social Sciences 

Psychology 18 

Law 13 

Movies 6 
3. Languages 

English 13 

Arabic 

French 7 
4. Personal Development 

Diction 4 

Charming 4 

Poetry 3 
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5. Health 


First Aid 3 
Medicinal and Aromatic Plants 3 
Patient Care (Schizophrene - Autism - Alzheimer) 2 


Answers of the question regarding the topics and courses of interest were also classified 
under five categories: technology, social sciences, languages, personal development and health. 
The table revealed that technology related topics and some of the fields in social sciences were 
pointed out by the participants more often than others. Adobe Products, coding and basic 
technology skills were the ones came to the fore than others in the technology category, while 
psychology and law were in social sciences. Unremarkably, learning English was another topic the 
participants were interested in learning. 


Conclusions and Recommendations 


MOOCs have been one of the hot topics over the last decade. Learners’ satisfaction and 
needs are very important to provide quality learning experience in MOOCs. This study focused on 
these two variables in the scope of Anadolu University’s MOOCs platform, AKADEMA. More 
precisely, the study intended to explore the learners’ satisfaction with different characteristics and 
components of the MOOCs and also their preferences of the topics they would like to learn in 
MOOCs. 

The study also investigated to what extent the courses met the learners’ expectations. The 
results have revealed that the courses in AKADEMA have met significant percent of the learners’ 
expectations, and a larger percent did not come across any problem during their study. From the 
data, it can be inferred that the participants had positive attitudes toward learning in MOOCs. 

The learners’ experiences in MOOCs was grouped under three categories: content and 
design, course site and media, and instructor behavior. The prominent characteristics or 
components of the AKADEMA MOOCs in terms of content and design were identified as the 
variety and up-to-dateness of the content in the courses, interactive learning design, as well as 
attention-grabbing characteristic of the content. Conole (2015) stated that one of the major motives 
for participating MOOCs is about personal development for self-fulfillment. The variety of courses 
may help them improve their skills or acquire new skills. This professional development 
opportunity as well as the certificate of completion they receive at the end might support for career 
development or switching to a new career. On the other hand, Vrasidas (2000) expressed the 
importance of content-to-learner interaction in any learning setting, while Tsang et al. (2014) and 
Zimmerman (2012) noted that this type of interaction may boost the motivation of the learners, 
reduce the drop-out rates and increase the chance of achieving the learning objectives in online 
learning. The current study uncovered a similar result that the participants indicated the interactive 
content of the courses as one of the satisfying components of the MOOCs. Another often indicated 
characteristics of the courses the participants liked was about appeal of the course contents. There 
were a number of studies in the literature (Aybek, 2017; Jacobs, 2013; Kirschner, 2012; Martin, 
2012; Zutshi, O’Hare & Rodafinos, 2013) emphasized the appealing of the course content among 
learners as one of the factors effecting the learners’ engagement and persistence. So, this study also 
supported the literature; namely the more appealing content we provide, the more learners engage 
with it and persist to complete the courses. 


97 


In terms of the course site and design theme, flexibility to reach the course site and 
resources, and the interactive videos provided to the learners were stepped forward as the most 
cited characteristics that the learners liked in the AKADEMA MOOCs. Since the leaners were in 
general lifelong learners whose major goals are related to personal development, it was important 
to access the course site and resources with different means, especially through mobile devices, 
and ways, including no login requirement. Flexibility, anyway, was considered as the key term for 
open education (Aydin, 2019; Cooke, 2018; Veletsianos & Houlden, 2019). Veletsinaos and 
Houlden (2019) analyzed the articles in the Journal of Distance Education and came up with six 
themes about flexibility in open and distance learning: the qualities of flexibility as affording 
“anytime, anyplace” learning; flexibility as pedagogy; liberatory or service-oriented aspects of 
flexibility; limitations of flexibility, especially in terms of technology, the constraints of time and 
space, as well as cultural differences; flexibility as a quality needed by instructors and instructional 
designers themselves; and critiques of flexibility as a concept. The current study considered the 
flexibility in terms of the first three themes of the Veletsinaos and Hounden’s study, namely the 
qualities of flexibility as affording “anytime, anyplace” learning; liberatory or service-oriented 
aspects of flexibility. The analyses have shown that the participants enjoyed the flexibility of 
learning anytime and anyplace accessing the learning environment and learning activities. 

Another important component of the AKADEMA MOOCs related to the course site and 
design theme was the interactivity required while watching the videos prepared by the instructors 
in-house to help learners acquire the competencies targeted. By looking at popular MOOCs in 
different platforms, one can easily infer that videos have come back big especially with the 
xMOOCs. Along with the practice, we have come across more and more studies on effective use 
of video as instructional material. So, literature provided beneficial insight about role of video in 
online learning and effective ways of producing as well as integrating them. For instance, studies 
(Bezerra & Silva, 2017; Clow, 2013; Wilkowski et al., 2014) shown that quality videos increased 
the learners’ motivation, and the exact opposite, poor quality videos caused loss of interest and 
motivation. Moreover, the studies in Turkey (Aybek, 2017; Horzum, 2016) revealed that only text- 
based content, which does not include videos, negatively affects participants' MOOC experiences 
and is even shown as a reason for dropping a course. The AKKADEMA courses have been 
providing interactive videos and also requiring some extra activities to turn the learners active 
watchers rather than passive receivers, such as asking questions and requiring the learners find their 
answers in the videos, or demanding the learners generate examples similar to those given in the 
videos, etc. It sounded like that the promotion of interactivity motivated the learners and supported 
their learning. 

Under the theme related to the ‘instructors’, the learners expressed their satisfaction with 
the instructors’ willingness and efforts to provide personalized feedback about their progress and 
their high level of subject-matter expertise. Previous studies, such as Horzum et al., (2016), have 
uncovered that insufficient feedback was listed as one of the reasons of dropping the MOOCs. 
Similarly, Bocchi et al., (2004), and Ivankova and Stick (2007) pointed out the significance of 
providing timely and sufficient feedback in open and distance learning, and how feedback help 
boosting the retention. In another study, Li et al. (2016) have also shown that instructors’ feedback 
supported the learners’ active engagement to the learning process and keep their focus on the 
learning activities. The MOOCs in AKADEMA platform required the instructors provide timely 
feedback. An automated messaging system was also built into the platform that informs the 
instructors about new and unanswered postings as well as some statistics about their actions in the 
course site via email. The instructors found it very beneficial for them to keep track of learners’ 
progress and their tasks in the courses. Furthermore, getting sufficient and timely feedback from 
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real instructors (not computer-generated interaction) possibly helped the learners develop a positive 
attitude about the instructors’ subject-matter expertise. A previous research finding indicated that 
instructor credibility has a significant positive effect on content area knowledge (Carr, Zube, & 
Dickens, 2013). In another study the researcher also found a significant relationship between 
learners’ perception of instructor credibility and the degree of their acquisition of learning 
outcomes in online learning (Vallade & Kaufmann, 2020). Our study also supported the literature 
about the relationship between learners’ perceptions with the credibility of instructors and their 
satisfaction from the courses. In the light of these, one can easily infer that personalized feedback 
and learners’ perception of instructors’ subject-matter expertise might be effective on the learners’ 
satisfaction and their retention in MOOCs. 

Another findings of the study supported the literature was about the learners’ preferences 
of the course topics for further learning: a big majority indicated technology related topics 
including some of the well-known productivity, such as Adobe Photoshop, Adobe Premier, and 
coding software. Since some of the learners attended the MOOCs for professional development 
and for new career opportunities, and technology skills definitely help individuals more forward in 
their career or get better jobs, this preference can be considered as acceptable and understandable. 
Meanwhile, one might find the finding related to the participants’ interest in psychology as 
extraordinary, but the field of psychology has recently been one of the top fields of study by the 
undergraduate candidates in Turkey (YOK, 2020) too. So, this finding also reflected the demand 
for psychology and related fields. Moreover, learning English language has always been one of the 
hot topics for decades in Turkey and most probably all around the world. So that, it was not a 
surprise to see this kind of a preference for AKADEMA. 

Overall, it was very remarkable to observe that the learners in MOOCs expressed their 
interest in different types of interactions, including peer, instructor and content. Literature on online 
learning (e.g.; Anderson, 2003; Moore, 2013; Miyazoe & Anderson, 2011; Rodriguez & Armellini 
2015) is full of studies express the importance and role of interaction on learning, retention, 
motivation, and etc. Similar relationship between interaction and learning is also documented for 
MOOCs too (e.g.; Cisel, 2018; Kasch, Van Rosmalen, & Kalz, 2020; Stracke, et al., 2018). 
However, one can easily claim that we need more studies on the different types and contexts of 
interaction especially in MOOCs to understand the phenomenon and provide better learning 
opportunities to all who wish to; and as Kasch et al (2020) stated in MOOCs we see thought- 
provoking examples of interaction but at the same time literature needs to elaborate interactions in 
MOOCs in order to improve their educational value and quality. This study provided a perspective 
in this sense about AKADEMA MOOCs and the participants’ preferences and satisfaction. On the 
other hand, the study should be conducted with the participation of more learners and also diverse 
data (qualitative and qualitative) on each components of the MOOCs should be collected to 
understand the interaction preferences as well as the most beneficial interaction types and activities. 
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Introduction 


Virtual technologies such as computer-assisted language learning (CALL), now available 
to most students, are ubiquitous in everyday life and are becoming increasingly essential to either 
online, or hybrid classes -in which students attend a traditional class twice a week but complete 
their homework virtually in an online platform. Particularly in universities in the United States, 
virtual platforms are increasingly used for teaching Spanish as a foreign language to students 
with varying backgrounds. As such, this research project proposes an approach to teaching 
grammar in an asynchronous setting, while considering common communicative goals that are 
established in a language course. Likewise, the proposed methods and strategies aim to offer an 
attractive language course that allows students to remotely learn and practice a language, while 
enhancing student engagement through interactive activities. 


Therefore, after reviewing several literature sources, methodologies for efficiently 
teaching a language online will be determined and shared in this research project. On one hand, 
the goal will be to demonstrate how effective certain strategies and tools can be, in order to 
design an interactive online course, which will allow all students to learn a language. On the 
other hand, challenges to consider while teaching a language online will also be discussed and 
analyzed. 

Although the efficiency of online language teaching is still questioned and at its infancy 
as could be noted in O’Dowd (2010) and Uschi (2003)’s articles, this research project will also 
provide with some data taken from Elementary Spanish courses that were taught at Stanly 
Community College. Data will support the grade increase that was noted in online and hybrid 
Spanish courses, after implementing new tools to increase student attention and engagement, and 
updating the design of Spanish courses according to the Quality Matters Standards. The research 
question that will be answered is the following: how can online language instructors design a 
Spanish course that fulfills all communicative and grammatical goals using certain tools and 
considering both design and student learning outcomes? 


Rationale and Purpose 


This project will discuss the way languages can be virtually taught efficiently to online 
students if certain factors and tools are implemented consistently and purposefully. For an online 
language course such as Spanish, tools have to be selected based on the three American Council 
on the Teaching of Foreign Languages (ACTFL) model standards (Cutshall, 2012), which 
include: 


- Interpersonal communication through speaking, writing, and listening activities, 
- Interpretive communication through listening, reading, interpreting and analyzing 
activities, 
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- Presentational communication through narrating, informing and explaining information 
through a variety of activities using the target language. 


Research regarding online language learning and the abilities for technology tools and 
CALL are showing evidence on how opinions can diverge regarding their efficiency to teach 
grammar and communication skills to the learners. Some articles from the reviewed literature 
show that an online language class can allow learners to study a language at their own pace and 
practice with both teachers and students at flexible times, if the teacher chooses the correct tools 
to implement in the course (Sun, 2014). 


On the other hand, teaching an online language course appears to be a controversial topic, 
as some authors such as Uschi (2003) still believe that technology has yet to prove to assess the 
full efficiency of successfully passing on knowledge and grammar concepts to learners, while 
others are fervent supporters of the quality of teaching through virtual platforms. In a higher 
education setting, three groups of agents must be considered when establishing online courses: 
the administrators, the teachers and the students. Administrators believe that online courses allow 
them to save money, staff and classroom space, while teachers are focused on the quality of 
teaching, whereas students may resent taking online courses (Uschi, 2003). Therefore, in this 
article, the purpose will be to discuss the fact that adopting certain strategies, in order to improve 
the selection of content and tools to include in virtual courses, can help language instructors have 
successful outcomes at the end of each semester. 


However, it is important for instructors to keep certain considerations in mind. 
Depending on the learners’ background and abilities, certain tools and strategies may be more 
appropriate than others in order to teach grammar outside of the traditional classroom. As both 
students and instructors may not have the opportunity to interact with each other as often as in a 
traditional classroom, presenting a variety of virtual activities will help students to stay engaged 
and understand the goals of language learning (Levy, Wang & Chen, 2009). Acquiring the 
grammar is an important component that cannot be neglected, as it is the foundation of language 
learning. Students can often forget that component as they already have established certain 
objectives upon completion of their Spanish courses. Motivations can vary, but in the majority of 
the introductory videos that students created at the beginning of the fall semester of 2019, they 
mentioned are professionally-oriented or grade-oriented. The goal for the instructor is to accept 
these motivations and objectives and make students work towards them. However, in order to 
efficiently do so, they have to show them that in this particular case, Spanish grammar is the base 
for any communicative goal, and learning a language is paramount to achieve their professional 
careers. 


Therefore, how can instructors successfully design a Spanish course that fulfills all 
communicative and grammatical goals, using certain tools and methodologies, and considering 
both design and student learning outcomes? 


Literature Review 

Colpaert (2006) reviews pedagogical implications based on a research study that 
demonstrates the increase of online interactivity with online language programs. The content 
delivery, role of teacher and pedagogy are discussed. First, the technology-driven approach 
attempts to be a pedagogy based on innovative tools. Then, the attribute-based approach analyzes 
the capabilities of this medium to potentially impact learning. The affordance-based approach 
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evaluates its potential to enhance the language learning and teaching processes. Finally, the 
pedagogy-based approach more strictly details the needed technology for teaching and learning. 
However, there is still a gap between language pedagogy and CALL tools. To bridge that gap, 
working with language instructors, instructional designers and other multidisciplinary individuals 
could be potential solutions. 

A variety of online tools can be used in order to train language instructors in order for 
them to be proficient and ready to teach online courses. As Levy, Wang, and Chen (2009) show, 
in order to train two language tutors, several online tools, platforms and a synchronous learning 
management system developed by the National Sun Yat-Sen University of Taiwan can be used to 
teach languages online efficiently. The implementations of the models used to train online tutors 
include the following learning concepts: learning as behavior, learning as construction of 
knowledge and meaning, and learning as a social practice, which are concepts that are paramount 
to then be able to facilitate the students’ learning approaches. The training integrated a variety of 
synchronous and asynchronous tools that allowed all professors from China and Australia to 
become more familiar with the tools before starting to teach the students. In addition, using any 
new material before class is highly recommended to ensure that it works accordingly. 

Through this training, it was noticed that learners should first feel confident by practicing 
using every feature of the learning management system (LMS) or any of the recommended tools 
by trying them out, assessing the language proficiency level in each class, in order to build 
familiarity and confidence among the learners. Through self-reflection and monitoring reports, 
learners were also able to interact with peers and share suggestions to the facilitator when 
needed. 

Additionally, in order to foster interaction among online language learners, O’ Dowd 
(2010) showed that in Spain, programs of online language learning through a system of e-tandem 
exchanges between some European institutions, who would write and correct each other (it was 
referred to as “telecollaboration”’) was seen as efficient. In some cases, teachers had played the 
role of facilitators, by monitoring between the students’ exchanges. Communications can take 
the form of videos, photos and messages. Online intercultural exchange with students from 
around the world is greatly increasing as it seems to be a great preparation before studying 
abroad. However, this practice involves both face-to-face and online interactions which, 
according to the author, are best in order to foster learning. Due to communication cultural 
differences, it may lack of authenticity without face-to-face interaction. 

In both face-to-face and online format, teachers change roles according to the constantly 
changing classroom settings. Nowadays through the Web 2.0, knowledge is now accessible to all 
learners and teachers (Senior, 2010). Based on the constructivist theory, knowledge is acquired 
socially through experience rather than discovered. In the case of language instructors, they have 
to be ready to teach and adopt different pedagogy as opposed to only focusing on constructivism 
or connectivity. Their teaching methods will be a blend of all theories. 

Within the classroom, teachers must ensure a sense of community and connectedness 
among the students. The instructor also has to build a relationship with students, no matter the 
grade. Establishing rapport can be done in a different class setting but is paramount to build a 
social environment, which will give the students a sense of comfort and engage them to work on 
the material in group for instance. LMS are great tools to allow students to remotely connect and 
the teachers play an important role as facilitators of these connections among students. Although 
traditional classes allow for more interaction, online courses can also be used as a social space if 
the instructor promotes collective group and learning. 
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In order to go from a virtual classroom to building an online learning community, learner 
behaviors are currently changing and will impact the way instructors teach (Sun, 2014). 
Depending on the level of expertise for online language teachers, not only technology skills must 
be taught to them but also pedagogy. For a novice online language instructor, these include 
strategies for online community building, socializing, facilitating communication competence, 
language learning theories and online language assessment. For an expert, they consist of 
creativity in using and adopting materials to create more materials to facilitate communicative 
competence and online interaction, facilitating online socialization and community building, and 
intuitive integration of assessments (Compton, 2009). Assessing and evaluating progress is an 
essential part of education and can be done by using the ACTFL guidelines. However, 
insufficient opportunities for students to speak the language through online tools makes it 
difficult to correctly assess oral language practice (Lin & Warschauer, 2015). 

Wang and Chen (2009) determined that synchronous meetings have been challenging as 
students cannot meet at the same time, or need to troubleshoot to fix technical issues at first. A 
traditional class cannot truly be replaced. Nonetheless, a virtual drop-in option has seemed to be 
more successful in one of their Elementary Chinese I class, as students could log in and practice 
with the teacher at any time. Learner participation in online language classes is always a priority 
as language learning is a skill-based rather than a content-based process. Constant synchronous 
interaction is needed to help learners practice with other peers. 

Nowadays, online learning is attracting a variety of students and in this article, Blake 
(2011) aims to show how the field of CALL and other computer-mediated communications can 
really facilitate online language learning (OLL). In this study, Blake chose to focus on web- 
facilitated, hybrid or blended, or fully virtual or online course. 

Often times, some concerns exist about OLL educational quality compared to traditional 
face-to-face courses. However, it is important to remember that traditional courses quality of 
teaching may also vary depending on the techniques and methods that are used, class sizes, and 
individual student attention. In fact, Grgurovic (2007) conducted a study that shows that students 
who took all their classes online performed better than their peers who took traditional courses 
due to the amount of time spent on them. Regarding CALL design, two areas have flourished in 
OLL: tutorial CALL, and social computing or computer-mediated communication (CMC). Often 
used to give grammar activities, there are also other tools recommended by Blake (2011), such as 
Quia and Hot Potato. However apart from grammar, focusing on lexicon and vocabulary through 
reading and listening comprehension assignments is also paramount. Nowadays, new tools are 
emerging rapidly due to the popularity of online courses. CALL such as Livemocha or Mango 
are constantly improving content delivering to better serve the students and increase language 
practice (Lee, 2016). 

On one hand, Intelligence CALL are used to provide feedback to students’ responses, and 
allow interactions through keeping track of their work and giving them suggestions. Such 
programs already exist for German, Portuguese and Japanese. Additionally, the use of 
asynchronous tools in an online course allows students to develop cognitive engagement, as they 
have more time to elaborate their responses, engage in the language which contributes to reduce 
anxiety (Lee, 2016). 

On the other hand, Social Computing CALL is another system that is learner-centered 
and allow both students and instructors to have both synchronous and asynchronous 
conversation. Social media means allow stimulation for students, however it awakens certain 
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concerns as personal and academic networks can be mixed together and the purpose could be 
lost. 

Mobile apps are also increasing as cellphones are popularly used and aim to help learners 
memorize vocabulary and sentences (Hockly, 2015). Apps like Duolingo for instance identify 
learners’ needs to progress in the language learning process based on computer-generated 
algorithms. After completing an activity for instance, learners will use points each day if they do 
not go back to practice it again. This is based on cognitive learning, as the brain will slowly 
forget certain information after not being used for an extended period of time. 

Additionally, studies have shown that videos appear to be as successful for students 
learning in a traditional face-to-face class, and students prefer to use the chat message in order to 
react about the content than a microphone. However, through the use of Visu, Guichon (2010) 
determined the limitations or catachreses than can exist for teachers while hosting a 
videoconference, such as the navigation between webpage, chat, Word pages, and images shared 
during the videoconference. Additionally, teachers must establish the time needed for each 
session, so they can evaluate and assess students efficiently. Both technological and pedagogical 
objectives have to be considered jointly, and not successively prior to hosting a session. 
Anticipating possible difficulties can save teachers and students time and energy, as visual 
materials can be provided before the session in one page so students can follow along easily 
without losing focus. 

Games are another way to stimulate language learning, as it has the potential to combine 
tutorial CALL with the attractive affordances of social computing. Certain videogames such as 
World of Warcraft allow different users from all over the world to interact and communicate 
while pursuing different quests. There are three different types of game: multiplayer, two- 
players, or single-player games. They have design experiences, which allow the game to be 
interactive and playful but this will be the main criteria for a learner/player to continue using it. 

Role-playing is another way to stimulate students’ attention by adopting a new identity 
while playing, which is what language instructors except the students to do when learning a 
foreign language. Students turn off their original language and culture in order to fully embrace 
the target language and its cultures. Blake (2011) mentions a variety of games can be used for 
language learning and have captivated students from the new generation as it has been 
documented. Ultimately, games and technology can be leveraged to connect traditional students 
as well so that their interests for the language does not fade away. 

Chun, Smith, and Kern (2016) talk about how technology and media influence context 
and forms of expression and communication. They intent to empower the learners so they can 
use the technology tools at their full potential. Often times, teachers encounter the following 
dilemma: technology can either be seen as extremely powerful to enhance intellectual capacity 
and creativity. On the other hand, there is the concern for some that technology could be 
detrimental to language learning. Tuoven (2000), as cited in Lee (2016), states that the lack of 
face-to-face communication or interaction and the isolation of students if learning alone with the 
computer can increase anxiety and decrease student learning process and productivity. 

Nonetheless, technology automatically influence on communicative purposes, which 
therefore can be used in order to learn and teach languages. Multimedia tools such as Camtasia 
that allows video capture to increase student engagement and interactions, collaborative and 
social tools such as social media are synchronous and asynchronous tools that can be used for 
communicative and semiotic purposes. They are designed as CMC. Games and applications are 
also available to students, teachers and researchers. To some extent, even online translators such 
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as Google Translate can be used effectively if the teachers offer guidance to students on how to 
use them, through literacy and creation of digital stories for instance. 

Colpaert (2006) conducted a research project based on the ADDIE model (analysis, 
design, development, implementation and evaluation), which helped demonstrate that often times 
in online language courses, conceptualization is the issue, not technology. Conceptualization is 
defined as the creation of a concept as an answer or solution to requirements. For instance, an 
interactive textbook, an adventure game, or a treasure hunt. Teachers should set the pedagogical 
goals before they pick their language methods and technology tools. They also have the roles of 
designers when it comes to planning, and contributors when it comes to assisting students. 

In a case study, Guichon (2010) talks about how the development of a desktop 
videoconferencing platform allow students to move from being novice in both language and 
technology and their difficulties in adapting, as well as the strategies they later developed. 
Desktop Videoconferencing (DVC) was used in this study to develop methodologies and 
strategies for online language learners. Both DVC and CALL allow for speaking assessment 
through synchronous and virtual meetings. Guichon (2009) identified three types of skills: socio- 
affective (that takes into account individual learning styles and intercultural differences, as well 
as the ability to establish a relationship with the learner), pedagogical (designing engaging task 
for online communication), and multimedia skills (which operates the appropriate tools and 
manages interactions through them). Hosting an online session also requires a planning time 
prior to the start of the meeting, as well as a reviewing phase after it has ended. 

Son (2011) points out the large variety of CALL as well as Internet tools and how they 
benefit online language learning. Several examples are given depending on the purpose 
attempted to be achieved in the class, such as communicative, resource sharing, learning 
management systems, web exercise creation, dictionaries and concorders, presentations and so 
forth. In order to pick the correct tool, instructors have to ask themselves who will use it, when, 
where, why and how will the CALL tool will be used. 

Nonetheless, Garrett (2009) points out that instructors must include guidelines for the 
learners to use it efficiently, so they can benefit from them. No assumption can be made and 
follow-up assignments should exist in order to assess the students. Therefore, the language 
instructor itself has to be skillful and knowledgeable regarding the new tool. Evaluating the 
choice of the tools and justify them in and outside of the classroom. 

Lund (2006) presents innovative communicative opportunities that emerged from 
teaching a language online, such as English in a Norwegian secondary school. Using the LMS, 
the study examines how social factors more than pedagogical ones influence his students’ 
interactions in order to favor communicative language practice. Not only language acquisition 
can be done in the classroom, but also in a social context such as in the community. Several 
practices that were noted in online courses as new opportunities have emerged through 
technology, and can be connected to didactics so that teachers can adapt their courses while 
following this cultural trend. 

In order to build meaningful activities for the learners, it is important to consider not only 
how, why and to whom we are building the activities but also where and when, whether it is 
synchronous or asynchronous. Didactics must take context into account. Between standardization 
and pluralism, between homogeneity and heterogeneity there is a third space where English as a 
Foreign Language didactics can emerge as a boundary object translating between different social 
worlds and discourses while maintaining consistency in the form of a shared goal (Star & 
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Griesemer, 1989). Although student participation and interaction are important, teachers also 
need to interact with them in order to remind students of their role as experts and referents. 

Online language practices can emerge from different contexts and settings. Therefore, it 
is the teacher’s role to insure “didacticizing” new virtual spaces, by taking into account historical 
and cultural contexts as well, and not restrain time and space to the classroom only. 

Harrison and Thomas (2009) have found that students presented with different tools will 
choose the ones that they find most appropriate for their goals, learning methods and style. This 
learner-centered approach helps foster learners’ creativity and will allow instructors to re-think 
their roles and teaching methods. 

Nonetheless, a challenge that exists in online language teaching is to set some rules for 
students to be able to communicate and respond to each other. For instance, Nunan (2002) 
compares the challenges and rewards of teaching online to Japanese and other international 
students as an L2 at a graduate level. A balance has to be found between teaching and 
facilitating, as well as including learner-centered activities to allow students to take control of 
their learning. Online language instructors also have to ensure that all students are participating 
actively, as some may be new to online learning and more reluctant to participate. In addition, it 
is paramount to ensure that students are familiar with the technology used in this classroom. 

Although in the study, Nunan (2002) underlines the convenience of the course as being 
able to be taken in different parts of the world, the student evaluations feedback reveals that 
students would prefer taking face-to-face classes than distance learning. On the other hand, the 
convenience for both teachers and students to be able to access the course from anywhere in the 
world was seen as an advantage. In fact, the creation of an online global community was also 
seen as a reward, as they had the opportunity to socialize, network and exchange tips. 

In this study, language instructors that were interviewed show that although they had 
great technological support by their institution, the pedagogical training to teaching online and 
hosting synchronous or asynchronous sessions were problematic. Teachers were eager to learn 
more about telecollaboration and integrating the online component to their classrooms. However 
as technical support does not seem to be available to support their project, teachers as very much 
laggard and would rather adopt this new teaching component after some improvement is ensured, 
and that their international peers who would be participating would agree in adopting this new 
teaching method as well. 

Another unclear aspect would be how to assess students in these online activities. For 
their normalization, informal feedback does not suffice in order to give a grade to a student. 
Therefore, a lot of skepticism still exists among European faculty as to integration of technology 
within the foreign language classroom. The normalization of an activity still needs to be 
developed in order to be fully integrated into the course curriculum and the syllabus. Partnership 
between the European institutions for telecollaboration also has to be settled first through 
exchange programs for instance such as Erasmus after online language instructors are trained 
accordingly. 

In online language classes, learners have to be emotionally and cognitively engaged by 
the instructors, who will provide personalized feedback according to the learners’ needs. A 
variety of synchronous and asynchronous tools to assess listening, reading, writing and speaking 
should be included in an online learning community. However, it is important to remember that 
interactions among students is also paramount for an online language course. 
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Therefore, controversies in online language teaching still exist among instructor, 
administrator and student standpoints (Uschi, 2003), as they state that technology cannot replace 
a traditional language class. 

In higher education, online teaching has been favored as administrators are interested in 
saving costs, time, staff and classroom space. However, this format shift may affect the quality 
of teaching, and students are often more skeptical about taking an online course. According to 
Noble (2001), education is an interpersonal affair and the use of technology interrupts it. 

For language teachers, at first, adding an online component led them to transfer part of 
the knowledge such as grammar and vocabulary to the online platform and dedicate more time 
for communicative activities in the classroom. Students appreciated the flexibility of doing 
homework online and at their own pace. On the other hand, administrators in higher education 
institutions decided to add online learning to the curriculum due to scarce resources, but without 
taking into consideration the possible impact on quality of teaching. Several teachers still lack 
expertise and support for online teaching. 

In addition, as new technology quickly appear and evolve, staying up-to-date and 
exploiting the tools at their full potential is a challenge and learners should be the priority as they 
will be the direct people to experience it. Adding technology to the language classroom has yet 
to improve the quality of interpersonal communication and include meaningful gap activities. 
Therefore, administrators do not save time or staff as students are more reluctant to taking online 
classes. 


Conclusion 

Despite the cost cutting that technology allows administrators and students to take 
advantage of whether it is classroom space, faculty stipend or tuition fees, online language 
teaching has yet to prove how effective completing online work is in order to become fully 
integrated in the curriculum. The pedagogy must be revisited by language instructors in order to 
maintain the quality of teaching that already exist in traditional language classes in order for their 
learners to feel a sense of community and allow communicative interactions. 

As Chun, Smith, and Kern (2016) showed in their study, the inclusion of technology in 
language courses will ultimately be subjective to each instructor. The tools that are implemented 
and their uses should align with the learning outcomes, as well as the learners’ interests, abilities, 
the available resources to each instructor and the culture of each institution. 

However, to support the efficiency of teaching a foreign language online, data was 
collected from beginning and intermediate classes at Stanly Community College. Previous to fall 
semester 2017, students did not yet have access to virtual tools in language classrooms. Between 
fall semester 2017 and fall semester 2018, online students began having access to more virtual 
tools with which they could interact orally, auditively, and in written form. Software we have 
discussed such as Zoom, Screen-Cast-O-Matic and Adobe Connect became available for 
instructors and students. This interactive use of technology has benefited students with the same 
quality of teaching students would get in a traditional classroom. Therefore, that can in part 
justify the growing curve that has been observed in specific language skills. 
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Table 1. Stanly Community College - Elementary Spanish 


Stanly Community College - Elementary Spanish Results 


Fall Semester 2017 
BB Spring Semester 2018 
Writing skills 


Oral skills 


Interpreting / 
listening skills 


Nevertheless, one limitation in presenting the data is that the percentage is calculated by 
the number of students in each class. This number can vary every semester depending on the 
demand. It is also important to note that every semester, students who register at the beginning 
level have a diverse profile. Some come with no previous knowledge of Spanish while others 
may already have some knowledge of the language from high school or from growing up in a 
bilingual climate. 


Table 2. Stanly Community College - Intermediate Spanish Results 


Stanly Community College - Intermediate Spanish Results 


Fall semester 2017 


BB Spring Semester 2018 
Writing skills 


Oral skills 


Interpreting / 
Listening skills 


As to the results taken from the intermediate class online, it is to consider that the 
majority was already familiar with the online format, as they had previously taken the 
elementary level with the same instructor. That could possibly justify the reason for which all 
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students from the fall 2017 and spring 2018 semesters had above average results in their oral 
skills, given that oral evaluations previously had a similar format. However, an increased curve 
between fall 2017 and spring 2018 was also observed in student’s writing, interpreting, and 
listening skills after implementing more virtual and interactive tools in the class. As a result, we 
are able to establish that even though a class is online, it is always important to show that a 
professor is present and available for students. The simple reality of being able to show one’s 
face through video creates more personal contact between students and their instructor. This 
leads to greater confidence between them, and the student’s autonomy and interest begin to 
grow. 

Although until now, few researches have been conducted on the efficiency of online 
language learning when integrated correctly, technology tools appear to be beneficial to OLL 
when they allow students to improve their listening, reading, writing and speaking skills and 
when instructors are able to assess their progress by providing substantial feedback and 
interaction (Blake, 2011). 
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Abstract 


This study was to investigate student perceptions of teaching presence in BlendFlex instruction 
as compared to face-to-face and online instructions. BlendFlex instruction offers additional 
flexibility for students to choose between delivery methods based on their needs and change it at 
any time. Selected items from the College Student Opinion Survey were used for measurement. 
The results revealed that student perceptions of teaching presence were significantly higher in 
BlendFlex instruction and face-to-face instruction than online instruction. 


Keywords: BlendFlex Instruction, Blended Instruction, Face-to-Face Instruction, Online 
Instruction, Teaching Presence 


Introduction 


Blended instruction has the advantages of both face-to-face and online instructions. In 
blended instruction, instructors provide content via a combination of face-to-face and online 
instructions, delivering between 30 percent and 79 percent of the course content online, with the 
remaining contact delivered through face-to-face or other non-Internet mediums (Gomes, 2014). 
Therefore, students can have face-to-face contact with their instructor and peers and also work in 
a self-paced mode that accommodates other scheduling needs (Currie, 2017). 

BlendFlex, a form of blended instruction that was pioneered predominantly by a technical 
college in Georgia, was provided to serve students who would not have local access to the 
instructional content but might not be comfortable with the self-paced format and demands of 
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online instruction (Central Georgia Technical College, n.d.; Durso, 2017; Lieberman, 2018; 
Quinn & Lee, 2016). BlendFlex instruction melds the face-to-face, online, and telepresence 
methods into a single methodology. Rather than being restricted to only one delivery method for 
the duration of a course, students registered for BlendFlex courses can, at any time, participate in 
as many, or as few, of the delivery methods as they wish. Instructors can assist students through 
multiple delivery methods in BlendFlex instruction. 

According to the Community of Inquiry (Col) framework, three core elements are 
essential to creating a meaningful educational experience: cognitive presence, social presence, 
and teaching presence (Garrison et al., 1999). Cognitive presence refers to one’s ability to 
construct meaning through interactions. Social presence relates to one’s ability to communicate 
directly and develop relationships based on the personality. Teaching presence is defined by the 
design, facilitation, and direction of both the social and cognitive aspects of instruction to 
provide meaningful learning in the course (Akyol et al., 2009). Sustained communication is 
important for achieving a cognitive presence in an instructional setting and can be difficult in 
face-to-face instruction, and proves to be even more difficult in online instruction. Learners in an 
instructional setting must be able to project their personalities into the community. Without the 
ability to project their personalities, learners in the online community may not view others as real 
people and therefore are unable to contribute to forming the core basis of a community where 
learners interact and build upon the knowledge and experience of others. To support the 
development of both cognitive and social presence, the function of teaching presence must be 
emphasized because it helps involve everyone in the community and produce a meaningful 
educational experience in a course (Garrison et al., 1999, 2010). Therefore, it is critical to 
investigate if BlendFlex instruction helps promote teaching presence as compared to face-to-face 
and online instructions. 


The Current Study 


BlendFlex instruction was pioneered predominantly by a technical college in Georgia 
(Central Georgia Technical College, n.d.; Durso, 2017; Lieberman, 2018; Quinn & Lee, 2016). It 
started on September 25, 2013, to provide students multiple delivery methods for healthcare 
training. Since its inception, BlendFlex instruction has expanded from healthcare-related courses 
to credit courses in other programs, non-credit adult education programs, and preparation classes 
for the GED® high school-equivalency test. This study focused on a math course due to the high 
number of sections offered by the college and the diverse student population registered for the 
course. 

To measure student perceptions of teaching presence, 10 items were selected from the 
College Student Opinion Survey. The selected items focused on student perceptions of the 
instructor's strategies related to “design and organization,” “facilitation,” “direct instruction,” and 
“others” teaching presence related measures. The reliability of the selected items was calculated 
using 107 students’ responses from MATH XXXxX courses in the fall semester of 2016 with a 
Cronbach’s Alpha of .97. From fall 2016 to spring 2019, 2,996 students actively registered for 
MATH XXXX that was offered in three different formats, BlendFlex, face-to-face, and online 
instructions. At the end of the course, students were invited to complete the survey. In total, 666 
students responded to the survey. The survey response rate was 22.23 %. One-way ANOVA and 
Games-Howell Post Hoc tests were used to analyze the survey results. 
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Results 


Data gathered from the College Student Opinion Survey consisted of responses from 
academic year (AY) 16-17 (fall 2016, spring 2017, and summer 2017), AY 17-18 (fall 2017, 
spring 2018, and summer 2018), AY 18-19 (fall 2018 and spring 2019 only). A total of 666 
responses were collected from students taking MATH 1012 courses during the defined timeline. 
Of those responses, 37 responses resulted from students participating in BlendFlex instruction, 
386 were attributed to students in face-to-face instruction, and 243 students in online instruction 
participated in the survey during the various terms (see Table 1). 


Table 1. Survey Responses by Academic Years 


Instruction Face-to- 
Years Covered BlendFlex Face Online 
AY 16-17 (Fall 2016, Spr 2017 and Smr 2017) 18 178 103 
AY 17-18 (Fall 2017, Spr 2018 and Smr 2018) 15 123 76 
AY 18-19 (Fall 2018 & Spring 2019) 4 85 64 
Subtotal 37 386 143 


Descriptive Statistics 


Based on the Col framework laid out by Akyol et al. (2009), teaching presence data were 
gathered via the ten selected items from the student opinion survey and divided into four 
categories: “design and organization,” “facilitation,” “direct instruction,” and “others.” Table 2 
shows the descriptive statistics for the survey results. Student responses of Strongly Agree were 
coded as 5 points, Agree coded as 4 points, No Opinion received a coding of 3 points, Disagree 
coded with 2 points, and Strongly Disagree was coded as 1 point. 


Table 2. Descriptive Statistics for Survey Results 


No. | Question Item BlendFlex FTF Online 
M (SD) M (SD) M (SD) 
Design & Organization 4.77(.71) | 4.67(.60) | 4.48 (81) 
1. | The instructor provided/presented a course 4.76 (.76) | 4.74(.59) | 4.55 (.84) 
syllabus outlining course requirements, 
course objectives, attendance/make-up work 
policies, and grading procedures. 
2. | The instructor taught the course objectives as | 4.78(.71) | 4.66 (.69) | 4.46 (.88) 
listed in the course syllabus. 
3. | The instructor followed the course schedule 4.78 (.71) | 4.68 (.65) | 4.49 (.89) 
or gave notice of any changes to the course 
schedule. 
4. | The instructor presented the material in a 4.76 (.72) | 4.63 (.78) | 4.43 (.91) 
clear and organized manner. 
Facilitation 4.73 (.74) | 4.61 (73) | 4.27 (.94) 
5. | The instructor encouraged students to ask 4.70 (.78) | 4.63 (.76) | 4.18 (1.07) 
questions and/or participate in class 
discussions. 
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6. | The instructor related course material to 4.76 (.72) | 4.60 (.78) | 4.37 (.93) 
practical application. 


Direct Instruction 4.70 (.72) | 4.55 (.80) | 4.22 (1.04) 


7. | The instructor provided suggestions on how 4.65 (.82) | 4.55 (.86) | 4.19 (1.09) 
to be successful in the course. 


8. | The instructor provided timely feedback on 4.76 (.72) | 4.55 (.82) | 4.25 (1.08) 
assignments and exams. 


Others 4.73 (73) | 4.56(.73) | 4.27097) 


9. | Adequate instructional time was given (in 4.78 (.71) | 4.60 (.79) | 4.37 (1.02) 
class or online) to cover the required 
material. 


10. | The instructor was available to assist me after | 4.68 (.78) | 4.53 (.79) | 4.17 (1.07) 
class hours (on-campus, electronically, 
and/or by phone). 


Note. The highlighted value is the highest mean between groups. 


The category of “design and organization” contained four questions (Q1, Q2, Q3, and Q4) 
and focused on information provided at the beginning of the course, the effectiveness of the 
course syllabus, the schedule of the course, and how well the instructor followed the initial 
information presented or provided a notice when changes were made. This category also 
evaluated the presentation of course materials. 

Students participating in BlendFlex instruction showed the highest perception score of 
teaching presence related to “design and organization” with a mean of 4.77 points (SD = .71). 
Face-to-face instruction-based courses scored slightly lower with a mean of 4.67 points (SD 
= .60). Students using online instruction indicated the lowest perception score of teaching 
presence (M = 4.48, SD = .81). 

“Facilitation” was the second categorical grouping used and consisted of two questions 
(Q5 and Q6) related to the instructor’s encouragement of queries from students, class 
participation, and relating the subject matter of the course to real-world examples. As with 
“design and organization,” students using BlendFlex instruction showed the highest perception 
score of teaching presence (M = 4.73, SD = .74). Face-to-face instruction scored slightly lower 
(M = 4.61, SD = .73) while online instruction showed the lowest perceived level of “facilitation” 
(M = 4.27, SD = .936). 

The “direct instruction” category consisted of two questions (Q7 and Q§8) related to 
suggestions on how to be successful in the course and timely feedback on assignments and 
exams. Students in BlendFlex instruction courses once again indicated the highest level of 
perceived teacher presence (MV = 4.70, SD = .721). Face-to-face instruction scored lower (M = 
4.55, SD = .80), while online instruction scored the lowest (M = 4.22, SD = 1.04). 

The last category, “others,” focused on the time allotted for instruction during class and 
the availability of the instructor outside of class. Once again, questions related to the “others” 
category revealed that classes taught using BlendFlex instruction had the highest perception 
score of teaching presence (M = 4.73, SD = .73). Face-to-face based instruction was in the 
middle (M = 4.56, SD = .726) while online instruction had the lowest perception score of 
teaching presence (M = 4.27, SD = .967). 

Students had the highest perception score of teaching presence in BlendFlex instruction 
in all categories and for every question. Some of the individual mean scores were not very far 
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apart. For example, when asked about the presentation of a course syllabus containing course 
requirements, course objectives, attendance/make-up work policies, and grading procedures, 
students in BlendFlex instruction courses (M = 4.76, SD = .76) responded similarly to those in 
face-to-face instruction (MV = 4.74, SD = .59). Other questions had considerable differences in the 
level of teaching presence perception. For example, BlendFlex instruction (M = 4.70, SD = .78) 
had a higher level of teaching presence according to respondents than online instruction (M = 
4.18, SD = 1.07) when asked about the instructor encouraging questions and class participation. 


One-Way ANOVA Test Results 


A one-way ANOVA test was used to test the impact of the instructional delivery method 
on student perceptions of teaching presence. As outlined in the descriptive analytics above, 10 
selected questions were divided into four categories: “design and organization,” “facilitation,” 
“direct instruction,” and “others.” “Design and organization” continued to focus on the initial 
presentation of information, following the presented course outline and schedule, communication 
of any changes, and material delivery throughout the course. “Facilitation” included questions 
related to encouraging questions, class participation, and relating the material to real-world 
examples. “Direct instruction” consisted of two questions related to the instructor providing 
suggestions on how to be successful in the course and providing feedback on assignments and 
tests. The “others” category was made up of two questions related to instructional time and 
instructor access outside of regular class hours. 

Questions | through 4 were categorized for “design and organization” as part of the Col 
framework. Specifically, these questions related to instructional delivery and relaying of course 
content. Q1 related to whether or not students perceived that their instructor provided a course 
syllabus outlining course objectives, attendance policies, makeup policies, and grading 
procedures, F(2, 663) = 5.73, p = .003 < .05. Q2 illustrated students’ perceptions of whether or 
not the instructor taught the course objectives outlined in the syllabus, F(2, 663) = 6.49, p = .002 
<.05. Q3 asked students if their instructor followed the schedule provided at the beginning of the 
term and whether or not the class was notified of any deviations from the originally presented 
schedule, F(2, 663) = 5.38, p = .005 < .05. Q4 examined whether or not students perceived a 
difference in teaching presence when asked if their instructors provided material in a clear and 
organized manner, F(2, 663) = 5.44, p = .005 < .05. As indicated by the data, there was a 
significant difference in student perceptions for questions 1, 2, 3, and 4 in the “design and 
organization” category (see Table 3). 

Question 5 and question 6 were grouped in the “facilitation” category. Q5 asked whether 
or not the instructor encouraged questions and engaged in-class participation, F(2, 663) = 20.48, 
p = .000 < .05. Q6 asked students if instructors related class material to real-world examples, F(2, 
663) = 7.31, p = .001 < .05. Both questions 5 and 6 illustrated significant differences in student 
perceptions of teaching presence between instructional delivery methods. Therefore, data showed 
a significant difference in student perceptions related to the “facilitation” category for the Col 
framework (see Table 3). 

Question 7 and question 8 were grouped for the “direct instruction” category. Q7 related 
to whether or not instructors provided suggestions on how to be successful in the course, F(2, 
663) = 11.92, p = .000 < .05. Q8 examined whether or not instructors provided timely feedback 
to assignments and tests, F(2, 663) = 10.29, p = .000 < .05. Question 7 and question 8 both 
showed significant differences in student perceptions of teaching presence between instructional 
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delivery methods (see Table 3). 

The last category of “others” consisted of two questions (Q9 and Q10) related to the time 
allowed for course material delivery and access to the instructor outside of class times. When 
asked about the time allowed for course material delivery, students indicated a perceived 
difference based on instructional delivery with F(2, 663) = 6.75, p = .001 < .05. Students also 
showed a difference in their perception of teaching presence using different instructional delivery 
methods with a result of F(2, 663) = 13.80, p = .000 < .05 (see Table 3). The “others” category 
represented the fourth and final piece to the Col framework. 


Table 3. One-Way ANOVA Test Results 
No. | Question Item Results 
Design & Organization 
1. | The instructor provided/presented a course syllabus | F(2, 663) = 5.73, p = .003 < .05 

outlining course requirements, course objectives, 

attendance/make-up work policies, and grading 
procedures. 

2. | The instructor taught the course objectives as listed | F(2, 663) = 6.49, p = .002 < .05 

in the course syllabus. 

3. | The instructor followed the course schedule or F(, 663) = 5.38, p = .005 < .05 

gave notice of any changes to the course schedule. 

4. | The instructor presented the material inaclear and | F(2, 663) = 5.44, p= .005 < .05 

organized manner. 

Facilitation 

5. | The instructor encouraged students to ask questions | F(2, 663) = 20.48, p = .000 < .05 

and/or participate in class discussions. 

6. | The instructor related course material to practical FQ, 663) = 7.31, p = .001 <.05 

application. 

Direct Instruction 

7. | The instructor provided suggestions on how to be | F(2, 663) = 11.92, p = .000 < .05 

successful in the course. 


8. | The instructor provided timely feedback on F(2, 663) = 10.29, p = .000 < .05 
assignments and exams. 
Others 


9. | Adequate instructional time was given (in class or | F(2, 663) = 6.75, p = .001 <.05 
online) to cover the required material. 
10. | The instructor was available to assist me after class | F(2, 663) = 13.80, p = .000 < .05 
hours (on-campus, electronically, and/or by phone). 


Games-Howell Post Hoc Test Results 


Games-Howell post hoc tests are used when data differ in sample sizes and there are not 
equal variances in each group (Starkweather, 2010). Since the College Student Opinion Survey 
resulted in differing sample sizes for students participating in courses using BlendFlex 
instruction (N =37), face-to-face (N = 386), and online instruction (N = 243), Games-Howell post 
hoc tests were performed. Table 4 illustrates the full results from the Games-Howell post hoc 
tests. 
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Games-Howell post hoc tests revealed that there were significant differences in perceived 
teaching presence in the “design and organization” category. All associated questions (Q1, Q2, 
Q3, and Q4) found significant differences in face-to-face instruction and online instruction. The 
student perception score of teaching presence related to “design and organization” was 
significantly higher in face-to-face instruction than online instruction. The tests also revealed 
significances in questions relating to teaching the course objectives (Q2) and material 
presentation (Q4) between BlendFlex instruction and online instruction. For these two items 
(course objectives and material presentation), the student perception score of teaching presence 
was significantly higher in BlendFlex instruction than online instruction. 

“Facilitation” related questions (Q5 and Q6) showed a significance in perceived teaching 
presence between instructional delivery methods. When asked about instructors encouraging 
students to ask questions and participate in class discussions, there was a difference in responses 
for BlendFlex instruction and online instruction (p = .002), and between face-to-face instruction 
and online instruction (p = .000). The student perception score of teaching presence was 
significantly higher in both BlendFlex instruction and face-to-face instruction than in online 
instruction. When asked about instructors relating course material to practical application, there 
was a difference in BlendFlex instruction and online instruction (p = .013), and between face-to- 
face instruction and online instruction (p = .004). The student perception score of teaching 
presence related to facilitation was significantly higher in both BlendFlex instruction and face-to- 
face instruction than online instruction. 

“Direct instruction” related questions (Q7 and Q8) showed similar results with 
differences in BlendFlex instruction and online instruction, with additional significance in face- 
to-face instruction and online instruction. Q7 related to instructors providing suggestions on how 
to be successful in the course. There were significant differences between BlendFlex instruction 
and online instruction (p = .011), and between face-to-face instruction and online instruction (p 
= .000). Q8 related to timely feedback received on assignments and exams. The result showed 
significant differences between BlendFlex instruction and online instruction (p = .001), and 
between face-to-face instruction and online instruction (p = .001). The student perception score 
of teaching presence related to direct instruction was significantly higher in both BlendFlex 
instruction and face-to-face instruction than in online instruction. 

The last category of teaching presence, “others,” had similar results with each question 
indicating the significant differences in instructional delivery methods on perceived teaching 
presence. The adequate instructional time between BlendFlex instruction and online instruction 
showed the same significance as face-to-face instruction and online instruction with p = .009. 
When asked about access to the instructor outside of regular class hours, there was a significant 
difference between BlendFlex instruction and online instruction (p = .003). There was also a 
significant difference between student perception scores of teaching presence between face-to- 
face instruction and online instruction (p = .000). 


Table 4. Games-Howell Post Hoc Tests Results 


No. | Question Item Results 

Design & Organization 

1. | The instructor 1) Significant difference between face-to-face instruction 
provided/presented a and online instruction (p = .007), M face-to-face > Montine 
course syllabus outlining 
course requirements, 
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course objectives, 
attendance/make-up 
work policies, and 
grading procedures. 

2. | The instructor taught the | 1) Significant difference between BlendFlex instruction 
course objectives as and online instruction (p = .039), MBlendFlex > Montine 
listed in the course 2) Significant difference between face-to-face instruction 
syllabus. and online instruction (p = .006), M face-to-face > Montine 

3. | The instructor followed 1) Significant difference between face-to-face instruction 
the course schedule or and online instruction (p = .017), Mface-to-face > Montine 
gave notice of any 
changes to the course 
schedule. 

4. | The instructor presented 1) Significant difference between BlendFlex instruction 
the material in a clear and online instruction (p = .042), MBlendFlex > Montine 
and organized manner. 2) Significant difference between face-to-face instruction 

and online instruction (p = .014), M face-to-face > Montine 

Facilitation 

5. | The instructor 1) Significant difference between BlendFlex instruction 
encouraged students to and online instruction (p = .002), MBlendFlex > Montline 
ask questions and/or 2) Significant difference between face-to-face instruction 
participate in class and online instruction (p = .000), M face-to-face > Montine 
discussions. 

6. | The instructor related 1) Significant difference between BlendFlex instruction 
course material to and online instruction (p = .013), MBlendFlex > Montline 
practical application. 2) Significant difference between face-to-face instruction 

and online instruction (p = .004), M face-to-face > Montine 

Direct Instruction 

7. | The instructor provided 1) Significant difference between BlendFlex instruction 
suggestions on how to be | and online instruction (p = .011), MBiendFiex > Montine 
successful in the course. 2) Significant difference between face-to-face instruction 

and online instruction (p = .000), Mface-to-face > Montine 

8. | The instructor provided 1) Significant difference between BlendFlex instruction 
timely feedback on and online instruction (p = .001), MBlendFlex > Montine 
assignments and exams. 2) Significant difference between face-to-face instruction 

and online instruction (p = .001), M face-to-face > Montine 


Continued on next page 
Others (Continued) 


9. 


Adequate instructional 
time was given (in class 
or online) to cover the 
required material. 


1) Significant difference between BlendFlex instruction 
and online instruction (p = .009), MBiendFlex > Montine 

2) Significant difference between face-to-face instruction 
and online instruction (p = .009), M face-to-face > Montine 


10. 


The instructor was 
available to assist me 
after class hours (on- 
campus, electronically, 


1) Significant difference between BlendFlex instruction 
and online instruction (p = .003), MBlendFlex > Montine 

2) Significant difference between face-to-face instruction 
and online instruction (p = .000), M face-to-face > Montine 
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and/or by phone). 


Discussion 


The results showed that the student perception scores of teaching presence were 
significantly higher in face-to-face instruction than online instruction for all survey questions. 
Except for survey questions one and three, the student perception scores of teaching presence 
were significantly higher in BlendFlex instruction than online instruction. The results proved the 
success of BlendFlex and, similarly, face-to-face instruction. Vaughan et al. (2013) stated, 
“teaching presence is enhanced when participants become more metacognitively aware and are 
encouraged to assume increasing responsibility and control of their learning” (p.13). Anderson et 
al. (2001) found that teaching in an online environment is influenced by the lack of non-verbal 
communication cues that exist in face-to-face instruction. BlendFlex instruction provides the 
immediate non-verbal cues that help to increase the perception of teaching presence but also 
allows students to take more control of their learning by providing multiple ways to consume 
course content. This could help explain why BlendFlex instruction consistently rated higher in 
perceived teaching presence than online instruction. 


Conclusions 


The purpose of this study was to examine student perceptions of teaching presence in 
BlendFlex instruction when compared to face-to-face instruction and online instruction. 
BlendFlex instruction incorporates the benefits of both face-to-face and online instructions, 
which helps engage students in a meaningful educational experience. The results of this study 
proved the success of BlendFlex and, similarly, face-to-face instruction. BlendFlex instruction 
seems to be a legitimate, and ultimately worthwhile, methodology for producing better results in 
improved perception of teaching presence among students than solely online instruction. To 
achieve a successful implementation of this new instruction, institutions must be willing to invest 
in technology, training, and other resources to establish best practices for ensuring both faculty 
and students are prepared to get the most benefit out of the BlendFlex instruction. Students will 
need considerable support, both before and after instruction begins, to understand how they can 
best utilize this instruction to achieve their educational goals. 
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Abstract 
Massive Open Online Courses (MOOCs) have been a popular venue for people to learn new 
knowledge, craft a new skill, or expand their professional profile (Pheatt, 2017; Hew & Cheung, 
2014). To date, very few studies have considered credential-based MOOCs. This exploratory 
study of eight MOOCSs representing 31 course runs investigated learner behavior within both 
open-enrollment and credential-based tracks to identify and correlate any learning patterns or 
trends. Findings suggest that the verified-track learners (n=5,117) have higher participation and 
passing rates than the audit-track learners (n=544,868). Based on a survival analysis, about 50% 
of the verified track persisted for 11 to 12 weeks. Using a multiple regression approach, 
persistence, gender and enrollment time period were the strongest predictors of the total grade. 
Finally, the study provides insights into student performance and identifies opportunities to 
better support learners through future research and course design in MOOCs. 


Keyword: MOOC, MicroMasters, Online learning, credential-based MOOC, behavioral data 


Introduction 

A Massive Open Online Course (MOOC) is an online course with the option of free, 
open registration and unlimited participation (McAuley et al., 2010). Anyone with internet 
access can enroll ina MOOC class to learn a new topic and grow their professional profile 
(Pheatt, 2017). According to a Class Central report in 2018, there were 101 million learners 
signed up for MOOCs, and over 900 universities were offering 11,400 MOOCs (Shah, 2018). 
Given their rapid expansion, educators and researchers are still skeptical of the effectiveness of 
MOOCs for the following reasons: 1. Low completion rate (3%-10%): MOOCs have a high 
dropout rate because learners have various learning goals, prior knowledge, skills and interests 
(McAuley et al., 2010); thus, not everyone aims to finish the course or earn a certificate (Pheatt, 
2017). 2. Lack of personalized feedback: current MOOC platforms utilize text-based discussion 
forums which are available for everyone. That makes it difficult to provide customized feedback 
for individual or small groups of learners (Yousef et al., 2014). Another reason for lacking 
individual feedback could be resulting from the large numbers of student enrollment and only a 
small amount of course staff is available to reply to questions (Daradoumis et al., 2013). 
3. Limited collaboration and social interaction with peers: unlike traditional classroom, the 
primary communication approach in MOOC is through text-based discussion forum (Onah, 
Sinclair & Boyatt, 2014). However, since students are not required to participate in the forum, 
only a small portion of students would post questions, share learning tips, or answer other 
students’ questions. The high dropout rate and open enrollment also result in a smaller number of 
students using the forum. With open enrollment, students could start the coursework at different 
times and that makes it hard for learners at different stages to engage in dynamic and 
spontaneous conversations with each other. Additionally, the graded assessments are usually 
designed as individual tasks, which further limits the opportunities for peer interaction (Gamage, 
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Fernando & Perera, 2015). 4. One-way delivery: a MOOC usually features a guided curriculum 
and pre-recorded video lectures. When students have a question during a lecture, they could not 
raise their hands to ask questions or have a theoretical debate like being in a traditional 
classroom. Instead, he or she could only rewatch the video again and again, with the hope that he 
or she could figure out the concepts eventually. If still confused, a student could post questions 
on the discussion forum and hopefully to get a reply from the course staff or classmates from a 
couple hours to a few days. In short, the one-way delivery style limits opportunities for students 
to interact with instructors (Gamage et al., 2015). 

Although educators and researchers have many concerns around MOOCs’ pedagogy 
design, technology resources and social dynamic, MOOCs continue to grow and expand rapidly. 
MOOC providers such as Coursera and edX began developing a new business model based on 
microcredentials because previous research suggested that learners are more willing to pay if 
they can earn a certificate upon completion (Eckstein, 2019; Young, 2013). A microcredential, 
defined by the National Education Association (n.d.), is a type of digital competency-based 
certification used in formal or informal professional learning programs. By the end of 2018, 630 
microcredentials in MOOCs are created, mostly offered by Coursera’s Specialization and edX’s 
Professional Certificate (Shah, 2019). 


What is a MicroMasters Program? 

A MicroMasters program is a type of credential-based MOOC offered by edX. Compared 
to open MOOCs, MicroMasters MOOCs require higher tuition fee, [D-verification, completion 
of graded assessments and a proctored final exam. The common topics of MicroMasters 
Programs include Business Management and Computer Science (Caudill, 2017). MicroMasters 
programs are often designed as a pathway to enroll in a University. For example, the Supply 
Chain Management MicroMasters Program offered by the Massachusetts Institute of Technology 
(MIT) allows students who successfully earn the credentials to enroll in a residential master’s 
degree program or convert credits to 20 other affiliated institutes (Kiers, 2016; MIT, n.d.). 

In the wide array of MOOC profiles, the credential-based MOOC is becoming a popular 
option for learners to pursue because the credentials are assessed and endorsed by a university 
certificate, so the credentials are more recognizable by potential employers (Terras and Ramsay, 
2016). However, little is known about the credential-based MOOCs, due to the following 
reasons: (1) not all universities have the resources to conduct learning analytics analyses (2) 
learner data is not easily accessible to researchers due to confidentiality (Almeda et al., 2018). 
Therefore, this study aims to explore learner characteristics and behaviors in eight credential- 
based MOOCs with an intention to identify learner behavioral pattern(s) within and between 
open MOOCs and credential-MOOCs. 

Literature Review 

Extensive MOOC research investigated learner dropout because it is a proxy for course 
completion (Reich, 2014). Machine learning (ML) algorithms are often utilized to conduct 
dropout modeling (Borrella, Caballero-Caballero, & Ponce-Cueto, 2019). The common ML 
algorithms are logistic regression, decision trees, random forests, support vector machines, 
neural networks and survival analysis; however, most of the literature suggests that these models 
have “similar predictive power” (Borrella et al., 2019, p3). Reich (2014) utilized Kaplan-Meier’s 
survival analysis to model learners’ attrition rates from nine HarvardX MOOCs. The survival 
analysis model considered learner persistence as a function based on the total number of days 
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students remained active in the course and dropout value. With the same perspective, this study 
also adopts Kaplan-Meier’s survival analysis to explore learners’ persistence and attrition rates. 

Learner’s gender, education levels, social-economic status, and proficiency in English are 
reported potential barriers to people’s understanding and accessing online information 
(Livingstone, Couvering, and Thumin, 2005). Since MOOCs attract learners with a variety of 
reasons ranging from preparation (Hew & Cheung, 2014), learner demographics and motivations 
could affect their behavior and level of participation (Rieber, 2017). This study was informed by 
the prior study to consider the demographic variables for the model analysis. 

Research Question 

This analysis aims to explore learner participation, behavior and characteristics in 
relationship to their learning performance between open-enrollment (audit track) versus paid- 
enrollment (verified track) in credential-based MOOCs. The research questions are listed as 
follows: 

RQI1: Do students in the verified track have higher participation rates for graded assignments 
(quizzes and projects) compared to their audit track peers? 

RQ2: Do students in the verified track have higher passing rates for graded assignments (total 
grade reports from quizzes, projects, and final exams) compared to their audit track peers? 
RQ3: What is the learner attrition rate for the verified track? 

RQ4: What factors predict learner grades for the verified track? 

The first two questions uncover differences or similarities between levels of interaction 
and academic performance between the audit and the verified tracks. The third and fourth 
questions illuminate the underlying mechanisms of learning patterns associated with academic 
performance. These findings should support future research design and identify opportunities to 
better engage learners in future MOOCs. 


Research Methods 
Data collection 

This study used data collected from two MicroMasters Program MOOCs, each consisting 
of four courses offered by an Ivy League school delivered through edX during 2017-2018. These 
courses were computer-science (CS) and business analytics(BA) related topics. Each course was 
12-week with an additional week for the final proctored exam, except one course was shorter 
(10-week). Learners could enroll in the audit track for free as an Open MOOC or pay to be in the 
verified track. The criteria to earn a course certificate included: paying a course fee, verifying 
identification, taking a final proctored exam, and achieving a total grade of 60% or above. 

The audit track learners received the same curriculum, such as lecture videos, and 
assessments, but the final exam was simplified. Specifically, the final exam for the audit track 
was similar to the weekly quizzes, while the final exam for the verified track MOOC was an 
online proctored comprehensive assessment. 

There were a total of 410,780 learners in the CS MicroMasters Program from 20 course 
runs and the BA MicroMasters Program had a total of 139,205 learners from 11 course runs(see 
table 1). No participants were recruited solely for the purpose of data collection. The learners’ 
gender breakdown in the CS MicroMasters Program were 65.9% male and 13.1% female. The 
BA MicroMasters Program had 56% male and 21 % female. 
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Methods 

For research questions | and 2, I compared the results between the audit and verified 
track learners using the Welch T-test. Due to the low participation and passing rates (less than 
10% and less than 5% respectively) of the audit track found in RQ1 and RQ2, I decided to 
review only the verified track data for the later questions. 

To examine the RQ3 on learner attrition, a nonparametric estimator of the survival 
function (Kaplan Meier method) is utilized to estimate and graph survival probabilities as a 
function of time (Min et al., 2011). Rstudio and the survival and survminer packages were used 
for analysis. Learner persistence was calculated based on learner’s first day of learning activity 
to the last day of learning activity (within the course beginning and end dates). Any activities 
beyond the course end date were not included in the analysis. 

For RQ4, multiple regression was conducted to investigate which factors were strong 
predictors of the total grade. The independent variables included: (1) student persistence (2) 
gender (3) educational levels (4) enrollment time period: early enrollment, (W1-W4), mid- 
enrollment (W5-W8), and late enrollment (W9-W12). Table 2 shows the descriptive data of the 
variables on the total grade. 


Results 

RQI1: Participation rate 

A weighted average of participation rate was calculated to produce accumulated results 
for the CS (CS101-104) MOOCs as well as the BA (BA101-104) MOOCs. Figure 1 shows the 
weighted learner participation rate in graded assignments (quizzes and projects) for the audit 
track and verified track respectively. In the CS MOOCs, the participation rate for the verified 
track in quizzes started at 70% and decreased to 35% by the end of the course. The participation 
rate for projects began at 40% but dropped to 11 % by the end of the course. Verified track 
learners in BA MOOCs followed a similar trend, with quiz participation rate starting at 70% but 
decreasing to 39%, and project participation rate falling from 61% to 28%. Conversely, the audit 
track exhibited a much lower participation rate. Figure 1 shows that their participation rate 
started from roughly 10%, then decreased to less than 1%. A Welch t-test analysis confirmed that 
verified track learner participation in quizzes and projects in both MicroMasters programs was 
significantly higher than those of the audit track (p<0.01). 
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CS 101-104 Quiz Participation Rates BA 101-104 Quiz Participation Rates 
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Figure 1 Weighted participation rate for quizzes and projects based on enrollment type in CS 
and BA MicroMasters Programs 


RQ2: Learner passing rate 

Table 1 shows that the passing rate for the verified track in the CS MicroMasters MOOCs 
was 33.9%, while the passing rate for the audit track was 0.14%. BA MicroMasters MOOCs, 
however, had a higher passing rate of 43.72% for the verified track, with a passing rate of 0.19% 
for audit track learners. On average, the audit track passing rate across eight MOOCs was 0.17% 
compared to verified track 38.81%. Based on Welch t-test results, it is concluded that the verified 
track passing rate is significantly higher from that of audit track (p<0.01). 


Table 1 Enrollment and Passing learners of CS and BA MOOCs 2017-2018 


Enrollment Passing Learners 
Total Verified Audit Total Verified Audit 
CS MOOCs 410,780 3,994 406,786 1,935 1,354 581 
(n=20 runs) (0.1%) (92.4%) (33.90%) (0.14%) 
BA MOOCs 139,205 1,123 138,082 752 491 261 
(n=11 runs) (0.8%) (99.2%) (43.72%) (0.19%) 
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RQ3: Attrition rate as learner persistence 

Attrition rate, as measured by a Kaplan-Meier survival analysis, was utilized to measure 
student dropout and academic persistence (Ascend Learning, 2012). Figures 2 and 3 showed the 
Kaplan-Meier survivor functions of CS MOOCs and BA MOOCs using learner active days in the 
course (academic persistence). Both CS and BA MOOC data showed a sharp rate of dropping 
out in the first few days, with attrition increasing steadily as the course continued. In CS 
MOOCs, about 50% of verified track learners dropped out by day 76, which was slightly less 
than 11 weeks. For BA MOOCs, 50% of verified track learners persisted up to day 85. 
Considering the total length of the course (13 weeks), the study results showed that about half of 
the verified track learners in the CS MOOCs persisted for 11 weeks; whereas 50% of the verified 
track in the BA MOOCs did not drop out. 
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Figure 2 Kaplan-Meier survivor functions Figure 3 Kaplan-Meier survivor functions for 
for CS MOOCs verified track learners BA MOOCs verified track learners (n=665) 
(n=3,258) 


130 


RQ4: Regression analysis of predictors of learner grades 

A multiple linear regression analysis was applied to model the predictors of total grade. 
Table 2 included the independent variables such as learner persistence, gender, educational 
level and enrollment time period. After examining the standard residual diagnostics, I 
determined that a quadratic specification of learner persistence was most appropriate for the 
analysis. 


Table 2: Descriptive Statistics of Independent Variables on Total Grades 


Independent Variables N M SD 


Learner Persistence 3,643 42 39 
(1~108 days) 


Gender 
Female 372 36 37 
Male 2,441 62 39 
Other 834 34 39 


Educational Level 


Post graduate 1,125 46 39 
Undergraduate 1,203 44 39 
High school 264 39 38 


Enrollment Time Period 


Early enrolled 3,137 43 39 
Mid enrolled 350 39 39 
Late enrolled 158 38 38 


Note: M and SD are used to represent the mean and standard deviation of the total grade 
(scaled from 0 to 100), respectively. 


Table 3 displayed the coefficient and nested F-tests of the taxonomy of four fitted 
multiple regression models. Model | had a predictor (learner persistence) with numeric value, 
while the other variables added into models 2, 3 and 4 represented vectors of dichotomous 
predictor variables. Because of this, their estimates reflected the difference between the observed 
variables and the reference categories. I interpreted the results of a nested F-test( ANOVA) to 
determine if including information about these categories improved model fit. I determine the 
effects of individual predictors by interpreting the magnitude and direction of the estimated 
coefficients, reported both in their native values and as effect sizes (Cohen’s d). 

Model 4 was chosen as the final model for the multiple regression analysis and its results 
indicated that, on average, controlling all other variables, students who persisted for 10 days 
longer will have final grade 4 points higher (d=0.10). In comparison to female students, a male 
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student on average, holding other variables constant, will perform 5.27 points better (d=0.14). In 
terms of student enrollment time period, the model suggested that students who joined the course 
during the middle of the course (between week 4 to week 8) (d=0.62) and late (between week 9 
to week 12) (d=0.7) performed better than those who enrolled early. 

In conclusion, learner persistence, gender and enrollment time were statistically 
significant in predicting total grade. Specifically, learner persistence was the strongest predictor 
in the study, followed by enrollment time period and then gender. 


Table 3. Multiple Regression of total grade on learner persistence, gender, educational level and 
enrollment time period. 


Model 
MI = M3 M4 
ves (n=3644) wee) (n=3640) (n=3638) 
(intercept) 1.84 -1.26*** -1.27*** -9,79%*** 
P (1.48) (1.61) (1.65) (1.63) 
Learner persistence 0.34*** 0.33*** 0.34*** 0:37** 
(0.07) (0.07) (0.07) (0.07) 
Learner persistence 0.00*** 0.00*** 0.00*** 0.01*** 
(0.00) (0.00) (0.00) (0.00) 
S.27*** 5.24*** S 2k 
Male (1.06) (1.22) (1.16) 
0.26 0.55 
Post graduate (1.36) (1.3) 
Undergraduate -0.18 -0.7 
(1.32) (1.25) 

: 24.19. *** 
Mid enrolled (1.71.) 
Late enrolled 38.15*** 

(2.44) 
0.41 0.41 0.41 0.47 


R? 


General Linear Hypothesis (GLH) Test on Model (1, 2), Model (2,3) and Model (3, 4) 


cf) Pi uiepateO B nate~o ROE P iipeapsiien= © 
[pene are =0 Beal 
Change in F-statistic 1270 24.89 0.06 195.79 
df 2 1 2 2 
p-value <0.001 <0.001 0.94 <0.001 
Decision Reject Hy Reject Hy Do not Reject Hp Reject Ho 


Notes: *p < .05; ** p< .01; *** p < .001, cells are estimates (standard error). 
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Discussion 


This study analyzed eight credential-based MOOCs with a total of 31 course runs over 2 
years of dataset. The results suggested that the verified track learners had, on average, a 40-60% 
participation rate in graded assignments (quizzes and projects), while the audit track learners had 
less than 10%. The verified track learners also had a roughly 40% passing rate (completing the 
course with a cumulative grade of 60% or higher), whereas the audit track learners had 17%. 
This data corresponded with previous MOOC studies recommending learners who paid to enroll 
in MOOCs had a stronger intention to complete the course and earn certificates (Reich, 2014; 
Koller, Ng & Chen, 2013). However, the passing rates of the verified track students in MOOCs 
was still extremely low compared to for-credit online learning environments (Almeda et al., 
2018), revealing serious deficiencies in MOOC learning environments. The study also found that 
about 50% of the verified track of the CS MOOCs dropped out by the 11th week (of 13); in the 
BA MOOCs, approximately 50% of learners did not not drop out. These mid- or late dropouts 
demonstrated greater interest and effort in completing their courses than early dropouts, but 
challenges such as difficult course content or personal factors (i.e. psychological issues, work or 
family obligations) may have prevented them from continuing. It is recommended that future 
research and course designers focus initially on investigating the causes of mid- or late dropouts 
over early dropouts, because those who left the course early may have just been browsing the 
course content. 

The preliminary descriptive data suggested that mid- and late- enrolled students had, on 
average, lower grades than early enrollers. However, the results of the multiple regression model 
showed that late enrollers actually performed better than early enrollers when controlling other 
variables. Intuitively, this relationship seemed contradictory. However, one reason could be that 
linear regression oversimplifies the intercorrelation between learner educational level, the 
enrollment time period, and total grades. For future research analysis, it is suggested to utilize 
more complex models (e.g., multi-level path analysis) and feature engineering of the variables to 
shed light on the relationship of these variables to grades. 

In addition to analyzing the strongest predictors of student grade, it is equally important 
to consider what can be done to support student success in MOOCs. Prior online learning 
literature asserts that utilizing self-regulatory learning (SRL) strategies in online learning 
environments has a positive effect on academic performance (Azevedo, 2005; Broadbent and 
Poon, 2015; Winters, Green, & Costich, 2008). Providing more scaffolding and feedback and 
assisting learners with developing self-regulatory strategies can increase learners’ motivation and 
learning performance (Schunk & Zimmerman, 1998). Lastly, social support such as increased 
peer interaction and smaller study groups could provide both cognitive and psychological 
support by providing a learning community (Effeney, Carroll & Bahr, 2013; Kellogg, Booth & 
Oliver, 2014). Unfortunately, the MOOCs included in this study provided only a small number 
of SRL supports on the discussion forums; there was no comprehensive and purposeful design to 
facilitate or teach SRL strategies. Although there are extensive studies regarding SRL in online 
learning, there have not been many experimental SRL studies conducted in the credential-based 
MOOC environment (AI-Freih, 2017; Almeda, 2018; Wong et al., 2019). With the global 
MOOC market size growing exponentially from USD 3.9 billion in 2018 to USD 20.8 billion by 
2023 (Researchandmarket, 2019) more attention from researchers and educators is required in 
order to address shortcomings in current MOOC design. The purpose of this study is to present 


133 


emerging learning challenges in credential-based MOOCs and provide recommendations for 
future research and improved instructional design. 
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Abstract 


The study was conducted to investigate the impact of a digital review tool had on 
students’ grades, learning motivation, and engagement. An exploratory cross-sectional study was 
performed with two groups of students. The Experimental Group was taught using the digital 
tool FlipQuiz whereas the Control Group was taught with the conventional method. Pretests, 
posttests, and surveys on the students’ learning motivation and engagement toward gamification 
in language learning were the instruments used in this study. The data of Posttests of both groups 
were analyzed using Paired Sample ¢ test. The results revealed no statistically significant 
differences with regard to FlipQuiz on English vocabulary exam scores. The data of two surveys 
were analyzed using descriptive analysis, parametric data analysis, and nonparametric data 
analysis. Findings indicated that there was a strong relationship among students in both groups 
regarding the importance of including technology in the classroom as well as the overall 
experience of utilizing FlipQuiz in learning English vocabulary. 

Introduction 


Students need to learn to use the tools as they are mandatory in today's digital world. 
Having an effective classroom with needed equipment such as digital tools and recourses, 
internet, and computers will make their learning process more attractive, engaging, and 
meaningful. Hence, students align with their teachers in terms of having access to this new 
technological world. Teachers in this point must be prepared to organize the new approaches of 
learning that will allow students to have equal access of using the technological resources. 
Students then will understand and retain more when their learning is relevant, engaging, and 
meaningful to their lives. Students may realize that integrating technology in their learning 
process will make them live in a multitasking, technology-driven, various, and energetic world, 
which might impact on their lives positively as today's innovative world requires people with 
enough awareness of using technology. 
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Purpose of the Study 


The researcher examined the online gameshow-style test review tool for the classroom 
called FlipQuiz, which teachers can employ in order to motivate their students and summarize 
the positive effects of reviewing learning content. Unlike other learning tools, FlipQuiz 
demonstrated its effectiveness in the learning process, which FlipQuiz requires verbal 
participation. FlipQuiz is a web application in which questions are reflected via projection 
device, verbal responses are received from students, feedback and evaluation is done online 
(Solmaz & Cetin, 2017). The study addressed the question of how FlipQuiz is applicable to 
learning English as a second language, especially with learning English vocabulary. The 
researcher also examined the students’ point of view toward the use of FlipQuiz in classroom. 

The study aimed to investigate the effects of FlipQuiz on students’ English vocabulary 
test scores as well as their learning motivation and engagement. The engagement focuses on their 
learning development in vocabulary knowledge while motivation demonstrates enthusiasm in 
learning (Wichadee & Pattanapichet, 2018). The study focused also at encouraging students’ 
engagement and boosting their motivation by using digital games such as FlipQuiz rather than 
the passive methods use in traditional classrooms, which is a teacher-directed style. 


Supporting Literature and Research Question 
Gamification in Learning 


In education, schools, universities, and educational organizations have become active 
users of technology. This leads the researcher to extend the use of technology tools in education, 
especially in teaching and learning English as a second/foreign language. Gamification is a 
learning method that attracts students’ attention in order to enhance their learning performance, 
motivation, and engagement. Solmaz and Cetin (2017) defined gamification as the use of game 
elements and methods in educational environments. The concept of game-based learning in 
education was formed based on gamification concept that was created from the positive 
contributions of elements in the games, for example, competition, challenge, points, leaderboard, 
nicknames, avatars etc. (Solmaz & Cetin, 2017). 

Despite the importance of gamification in education today, there are still disagreements 
on the effectiveness of gamification on student learning. However, many researchers have 
studied the gamification and its effectiveness for learning and education and found a positive 
correlation between gamification and expected outcomes. One of the most important outcomes in 
learning and education is learning achievements (Kim S., Song K., Lockee B., Burton J., 2018). 
Many researchers indicated that gamification for learning can develop achievement of the 
learners. In the study by Kim and others (2018), higher order thinking skills, declarative 
knowledge and procedural knowledge, and test performance in the schools were improved by 
gamification. Psychological and behavioral changes, in addition, were noticed in other research 
findings. Hence, students' and learners' motivation and engagement were fostered through 
gamified learning environments, Kumar and Khurana (2012). 


Solmaz and Cetin (2017) indicated that students had positive attitudes toward 


gamification. Students points of view showed that gamification activities improved motivation, 
learning and fun and decreased test anxiety. Students showed a positive attitude when they 
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engaged in a game-based learning environment. Hanus and Fox (2015) supported this argument 
by stating that utilizing games in education have advantages. Many game design mechanics 
illustrated success in educational environments, such as play again and recoverable errors. 
Students' engagement made them study and interact without fear. Hence, playing and integrating 
online digital learning tools or platforms led to the positive effect on students’ performance. For 
example, an online digital platform called Kahoot improved students' motivation and satisfaction 
positively (Medina & Hurtado, 2017). 


Students' motivation and engagement are the main reasons for applying technology tools 
in learning process. Many studies indicated that games helped motivate students to learn (Hanus 
& Fox, 2015; Heaslip, Donovan, & Cullen, 2014; Medina & Hurtade, 2017; Solmaz & Cetin, 
2017; Wichadee & Pattanapichet, 2018). The lack of students' performance with learning English 
vocabulary allowed the researcher to investigate this problem with the use of a game-review 
digital tool called FlipQuiz. Similar digital tools such as Kahoot, Socrative, Quizlet etc. showed 
positive results in respect to students' motivation, engagement, and academic performance 
(Solmaz & Cetin, 2017). 


Research Question 


Does the use of FlipQuiz impact vocabulary test scores of 7 grade students in a Saudi Arabia 
English learning classroom? 


Research Design and Methodology 


Participants were selected from one of Saudi middle public schools with students enrolled 
in an English class. All participants were enrolled in the same English course. The aimed grade 
for the study is 7 grade that has two classrooms with almost equal number of students in each 
class. Thus, one class is the Control Group and the other is the Experimental Group. 

Participants in the Control Group were taught the English vocabulary lesson in a 
traditional classroom learning using Flashcards and Handouts at the end of the class period. 
Participants in the Experimental Group were taught the English vocabulary lesson in an advance 
learning classroom with technology equipment including an LCD projector, teacher computer, 
loudspeakers, and a high-speed Internet. The English teacher have employed 'FlipQuiz', which is 
gameshow-style boards for test reviews in the classroom, during studying the assigned unit. 
FlipQuiz used to review the given vocabulary lessons at the end of each class period. Then 
participants in both groups have completed posttest exams, in order to evaluate and compare 
their scores, at the end of the unit. McDaniel et al. (as cited in Iwamoto, Hargis, Taitano & 
Vuong, 2017) noted that, experiments that were done in 7" and 8" grade science classrooms 
applied different quiz and exam items as opposed to earlier reported experiments where 
questions were the same with random assignment of multiple-choice items. 


This quantitative research study is an experimental design that employed pretest and 
posttest designs (Experimental Group) and (Control Group) in order to compare the outcomes, 
and a survey used a Likert scale. The survey was given to students to see their opinion toward 
learning with the use of FlipQuiz as well as evaluated the use of FlipQuiz on their learning 
motivation and engagement (Iwamoto, Hargis, Taitano, & Vuong, 2017; Medina & Hurtado, 
2017; Wichadee & Pattanapichet, 2018). 
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Participants 


In this exploratory cross-sectional study, the sample size was 51 Saudi middle school 
students from Al-Asamelah Intermediate and Secondary School a public school located in Jazan 
region, Saudi Arabia. Data were collected during the Spring semester from January 2019 —May 
2019 for two weeks. Starting on March 10" and ending on March 25". Participants enrolled in 
7 grade English language course ages 12-13 (100% males). In the present study, convenience 
sample was used. Students in the Experimental Group used (FlipQuiz), students in Control 
Group used (Flashcards and Handouts). Both groups were taught using the same learning content 
and used review tools in the last ten minutes of class period. Survey and pretest posttest data 
were analyzed. 


Procedure 


The researcher applied the study with 7" grade students in Saudi Arabia. Students prior 
the intervention were required to provide a consent form signed by their parents in which they 
would have the option to participate in survey completion. The Pretest for Experimental Group 
and Control Group were conducted in order to evaluate the test scores average among students in 
both groups. The pre-test was a Paper-Based English vocabulary test. Its layout (10 questions- 
multiple choice exam). The total score of the exam was 20 points/scores, each question equal 2 
points / scores. 

After that, for two weeks, the researcher used FlipQuiz for reviewing English vocabulary 
lessons with the students in the Experimental Group. While students in the Control Group were 
taught with the conventional method, therefore, students in control group were taught by using 
FlipQuiz if needed following the study. Both groups were taught using the same content of 
English lessons. 


After completing the assigned unit, the researcher conducted the Posttest, which was 
Paper-Based English vocabulary test. It was the same design of the Pretest exam. As soon as the 
Posttest conducted, two printed surveys were given to the students in both groups Experimental 
Group and Control Group in order to evaluate their opinions of studying with the use of FlipQuiz 
and conventional method as well as their learning motivation and engagement. The data were 
analyzed using (descriptive analysis, parametric data analysis, and nonparametric data analysis). 


Results 


A paired-samples ¢ test was conducted to evaluate the impact of FlipQuiz on student’s 
English vocabulary test scores. There was no statistically significant difference between Control 
Group (M= 16.46, SD= 4.71). Experimental Group (M=17.6, SD= 4.24) conditions; t (24)=.76, 
p=.45. These results showed that both groups had no significant difference in posttest English 
vocabulary exam. 
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Paired Samples Test 


Paired Differences 


95% Confidence Interval of the 
Std. Error Difference 
Mean Std. Deviation Mean Lower Upper t df Sig. (2-tailed) 


Pair1 Posttest (Experimental 1.040 6.810 1.362 -1.771 3.851 764 24 453 
group) - Posttest (Control 


group) 


Paired Samples Statistics 


Std. Error 
Mean N Std. Deviation Mean 
Pair 1 Posttest (EG) 17.60 25 4.243 .849 
Posttest (CG) 16.46 26 4.709 .924 


Students in both Experimental Group with 96 % and Control Group with 68 % of the 
total percentage reported that there is a positive relationship regarding the use of technology and 
its importance for students in learning English vocabulary. This demonstrate that students were 
excited to experience the use of gamification in education. Students in Experimental Group used 
FlipQuiz in an effective way, which impact on the survey answers by stating that it is very 
important to integrate technology in the classroom. Only 4 % total percentage of participants 
replied with not important to integrate technology in the classroom. It can be seen also in the 
Control Group that 32 % total percentage of participants stated that including technology in the 
classroom is not important. 


The importance of integrating technology in the classroom The importance of including technology in classroom 


d enga: 

mVery important m Not important rwitht _ = 
percemiaye. wre 0170 total percentage of students in the Control Group want to try studying 
English with the use of technology rather than the traditional way of teaching. In terms of 
engaging technology in educational environment, Solmaz and Cetin (2017) indicated that game- 
based learning environment showed a positive attitude. This supports also the research by Kim 
and others (2018) indicated that one of the most important outcomes in learning and education is 
learning achievements, which results by motivation and engagement. 


m= Very important m Not important 
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Students enjoyed learning English vocabulary using FlipQuiz | want to try learning English with the use of technology 
rather than using flashcards and handouts 


12% 


88% 


mAgree mDisagree mAgree m Disagree 


Overall students' experience illustrated that students in Experimental Group has an 
excellent impact regarding learning English vocabulary using FlipQuiz. These results support 
other studies in terms of the positive effects of using gamification in learning (Hanus & Fox, 
2015; Heaslip, Donovan, & Cullen, 2014; Medina & Hurtade, 2017; Solmaz & Cetin, 2017; 
Wichadee & Pattanapichet, 2018). Integrating games in learning process has advantages, and 
many game design mechanics demonstrated success in educational environments, play again, 
making errors recoverable (Hanus & Fox, 2015). 


Overall scores between both groups 
Overall students’ experience 


” 


Excellent Poor 
= Learning English vocabulary using FlipQuiz 
mLearning English vocabulary using flashcards and handouts 


= Posttest Experimental Group = Posttest Control Group 


Discussions 


Learning that includes fun activities in the course outline create a good atmosphere for 
the students to gain knowledge. It generates more achievement and motivation to the students 
that are engage in it compared to those that are not. Therefore, FlipQuiz and other similar game 
review tools are good digital games, which can be used to increase the students’ interest in 
learning language and make it more fun. The results of the research showed that the FlipQuiz has 
a significant effect on student motivation and engagement, especially in the English vocabulary 
lessons. These results were consistent with participants’ motivation and engagment in terms of 
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using a game-based learning as described. In addition, many studies have illustrated that the use 
of gamification in learning environments positively influences student motivation, interest, and 
active participationby (Hanus & Fox, 2015; Heaslip, Donovan, & Cullen, 2014; Medina & 
Hurtade, 2017; Solmaz & Cetin, 2017; Wichadee & Pattanapichet, 2018). 


In this experiment, findings of posttest exams for Experimental Group and Control Group 
indicate that there was no significant difference among students’ English vocabulary exam 
scores. The surveys demonstrated a strong relationship among students in both groups regarding 
the importance of including technology in the classroom. Therefore, students in the Experimental 
Group enjoyed their experience with the use of FlipQuiz as well as students in the Control Group 
who want to try learning English with the use of technology. Findings also indicated that the 
learning experience of students in Experimental Group with using FlipQuiz was excellent with 
100 % survey answers. On the other hand, students in the Control Group survey answers showed 
that 88% of students found flashcards and handouts excellent to be used in learning English 
vocabulary, while 12% of them found that flashcards and handouts poor and not helpful. In a 
similar research study by Pektas and Kepceoglu (2019), participants showed that they were 
happy to be involved in the gamification application integration, they enjoyed it, they had an 
effective and lasting learning experience, they enhanced their success through positive 
competition, and their interests and motivations improved. 


The results of this study compared to others showed that there is a relationship between 
students' motivation and engagement as well as improving in terms of exam scores. Pektas and 
Kepceoglu (2019) indicated that teachers who participated in their study have impacted 
positively when using gamification with their students. They noticed an improvement in 
motivation, saving time, and preventing cheating, as well as limitations such as difficulty in 
classroom management and technological problems. In terms of assessment of instruction, 
participants also suggested to use gamification applications. 


It can be suggested that game-based review tools like FlipQuiz have the potential to 
enhance and improve English vocabulary examination scores at 7" grade level. Students in the 
experimental group felt positive about their experience towards the use of FlipQuiz. The findings 
of this study also suggested that including technology in the classroom, creating a fun and 
engaging, and motivating environment can support academic performance, which reflects on 
students' exam scores positively. Students will learn what excites them. If a student cares about 
what he is introduced to, he will be motivated to learn. 


Limitations and Future Research 


The main limitation of this study was the time period of the study. Within two weeks, it 
could be difficult to calculate any statistically significant results. In addition, the sample was not 
diverse in regard to age, gender and level of education. Students used to study with the 
traditional way of teaching, thus they would be more likely to prefer the traditional methods they 
are already familiar with. Therefore, the results of the study would be heavily influenced. One 
more limitation is that some students in the Experimental Group might not have had the 
opportunity to fully participate in the study because of shyness or hesitation, as FlipQuiz requires 
verbal participation. This means that motivating and engaging students during the class period is 
necessary. In this study, FlipQuiz was only applied in an English class with the vocabulary skill. 
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Future research in a variety of subject areas needs to be accomplished in order to see the wide- 
ranging advantages and disadvantages of FlipQuiz. 
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Abstract 


Despite the last 40 years of research showing that computer-aided diagramming tools 
improve learning, little research reveal the cognitive processes that explain how diagramming 
tools and specific features of the tools affect learning. This study developed a tool that 
students used to diagram and analyze arguments as it mined students’ actions. The mined 
data was used to develop algorithms to measure the use of backward, forward, breadth, 
depth-first reasoning. Regression models were compared to identify which algorithm 
produced measures that best predicted diagram scores, and to determine the relative impact 
of each reasoning process on diagram scores. The findings show that observing the 
placement the first five nodes moved on screen in relation to the location of the previously 
moved node provides sufficient data to generate backward/forward and breadth/depth-first 
ratio scores that predict scores, while individual frequency counts of each process do not 
predict scores. The best-fit regression model using the ratio scores show that students using 
more backward and depth-first processing construct diagrams with higher scores and greater 
depth of analysis. This study presents new tools, methods, and new lines of inquiry to 
advance research on ways to integrate analytics into diagramming tools. 


Introduction 


The ability to evaluate arguments is a critical skill for decision-making and problem 
solving (Larson, Britt, & Kurby, 2009). Furthermore, the ability to combine arguments and 
counter-arguments are also essential for constructing coherent and persuasive claims and 
propositions (Mateos et al., 2018). To develop these abilities, students can be taught the skills 
of argument analysis and apply them across domains and disciplines (Cottrell, 2017; 
Davies, 2006; Moore, 2004). Argument analysis is the study of logical relationships 
between propositions presented in an argument (which can be mutually supporting or 
opposing claims) for building and evaluating premises used to support a conclusion 
(Toulmin, 1958). In argument analysis, students identify the functional roles of each 
proposition (i.e., conclusion, major premise, co-premise, minor premises, counterarguments), 
analyze the hierarchical relationships among propositions (i.e., levels of premise), and 
evaluate the quality and line of reasoning represented in a chain of premises linked to each 
major premise. Argumentation can serve as a means to test uncertainties, extract meaning, 
achieve understanding, and examine complex ill-structured problems and issues (Jonassen & 
Kim, 2010; Kuhn, 1993). Identifying and evaluating the reasons to support a conclusion is a 
fundamental part of human reasoning and decision-making (Mercier & Sperber, 2011). 
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Given that arguments are difficult to analyze, computer-aided argument mapping 
(CAAM) tools have been developed (Davies, Barnett, & van Gelder, 2019) to provide a 
visual method of identifying inferential relationships and evaluating the structural soundness 
of arguments (Braak, 2006; Davies, 2011b). Diagrams reduce cognitive load, make 
relationships more concrete, and facilitates the analysis and interpretation of complex ideas 
(Kirschner, Shum, & Carr, 2003; Novak & Cafias, 2007). Argument Mapper (Wright et al., 
2017), GAIL (Green, 2017), and Rationale (van Gelder, 2007) are just a few examples of 
these tools. Some tools like REASON (ThinkReliability, 2007) prescribes specific logic rules 
such as backward reasoning and the goal-driven approach to guide users through the 
argument diagramming process (Sharma, 2012). Research on the efficacy of diagramming 
tools shows that they improve learning with moderate to large effect sizes (Schroeder et al., 
2017) and helps students achieve significant gains in critical thinking skills measured with 
standardized instruments (Eftekhari at al., 2016; van Gelder, 2015). However, the majority of 
the studies compare CAAM with traditional treatments where features of the treatment are 
often unspecified, thus pose as possible confounding variables (Schroeder et al., 2017). Very 
little of the research reveals the cognitive processes students perform while constructing 
diagrams and how specific features of the tools affect the cognitive process and resulting 
learning outcomes. Identifying the cognitive processes used to construct better diagrams is 
necessary to determine how diagramming tools can be improved to increase learning. 

One reason as to why little research has been done on the cognitive processes used to 
construct diagrams is that coding video recordings of such behaviors is highly complex. Few 
if any CAAM applications chronologically log each action performed on a diagram. The few 
studies that examined specific processes are qualitative case studies. Kim (2016) found that 
all five experts (persons with prior training on analyzing arguments) and four novices used 
more breadth-first than depth-first process, with one novice using both processes. A depth- 
first process (Figure 1) is performed, for example, when premise 2 is linked to premise 1, 
followed immediately by linking 3 to 2. In contrast, a breadth-first process takes place when 
premise 2 is linked to premise | followed immediately by linking premise 4 to premise 1 
(premises 2 and 4 being at the same level). The study also found that four of the five experts 
used a backward process to construct the diagram, while one novice used the backward 
process and four novices used both processes. The backward process is used when premise 2 
is linked to premise 1, followed immediately by linking premise 3 to premise 2. Performing 
two backward or two forward processes in succession produces one depth-first process. 


Depth-First Process Breadth-First Process 
| Claim | Claim 
(1 ] 5 fey) ea 
x ™ Ye . 
2)(4) LejJlsJ L3sJL4J LSJL8J 
y ra x ; A ; z 
L3 J L7 J L7 J 8 | 
Construct the left sub-tree first, Construct one level of child 
then construct the right sub-tree nodes first, then construct next 
level of child nodes, and so on 


Figure 1. Illustration of the depth-first and breadth-first process 
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Some studies examined how these processes are used to create causal diagrams for 
identifying the root causes of a problem and identifying the chain of events that linking root 
causes to outcomes. Video analysis of students individually working on causal diagrams 
reveals that the most accurate causal diagrams are produced using a backward process and a 
breadth-first process when positioning events in the causal diagram (Shin, 2016). Working in 
two student groups, Lee (2012) found that the most accurate causal diagram produced among 
three groups used the backward approach to link together the events that contribute to the 
outcome, while the other two groups show no tendency to use any one specific approach. In 
addition to these findings, Jeong & Lee (2012) found that placing the outcome node closest 
to the right edge of the screen and placing causal links that primarily and sequentially flow 
(or point) from left to right produces the most accurate causal diagrams. 

These prior studies show that using the backward and breadth-first process to identify 
relationships between the claim and premises can be more effective than using the forward 
and depth-first process. However, these prior findings relied on manual observation and 
coding of video recordings to identify diagramming processes, which is prone to error and 
requires the use of complex coding schemes and rigorous training between coders to achieve 
inter-rater reliability (Shin, 2016). Furthermore, the prior studies (Kim, 2015; Lee, 2012; 
Shin 2016) focused on identifying overall behavioral tendencies by categorically determining 
which process is used by each student when most students do not use any one process 
exclusively. As a result, prior studies did not analyze the number of times students use the 
backward process relative to the number of times they use the forward process, and hence, 
did not compare the relative impact of backward/forward versus breath/depth-first processes. 

To develop diagramming tools that can assess and guide students through the reasoning 
processes, this study developed the j;MAP tool to mine and log behaviors performed while 
constructing diagrams and created algorithms using log data to operationalize and measure 
the number of times each student performs a backward, forward, breadth-first, and depth-first 
process. Using the frequency counts to compute ratios on the number of times each student 
uses backward vs. forward (B/F ratio) and breadth- vs. depth-first process (BR/D ratio), this 
study analyzes the ratios using linear regression to address the following research questions: 


1. Which algorithms produce ratio scores that best predict students’ diagram scores? 

2. Does the use of backward and breadth-first processing produce higher quality 
diagrams as shown in prior studies? 

3. Which of the processes (backward vs. breadth-first) has a greater impact on scores? 


Method 
Participants 
The participants in this study were 17 Masters students in an online course on computer- 
supported collaborative learning at a large Southeastern university (7 male, 10 female, ages 
ranging from 22 to 55 years), majoring in learning technology and instructional design. 


Procedures and Materials 

The participants reviewed arguments produced by students in another prior course that 
participated in an online debate to support and oppose the claim “One’s choice of media 
significantly affects learning”. The students in this study were instructed to create an 
argument diagram that links the major premises to the claim (a claim that was familiar to all 
the students in the course and master’s program) and to identify and link together the chain of 
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premises supporting the claim. Students viewed a 43-second video on how to position a 
claim and a premise in jMAP (Figure 2) and how to insert a link to connect the premise to the 
claim. In jMAP, students can move and position the nodes containing a premise or claim by 
dragging and dropping and can manually insert links to connect related nodes at any time of 
their choosing. Making the insertion of links manual (as opposed to automated, as is the case 
in many diagramming applications) gives students the option to add links later so that they 
can easily re-arrange nodes to explore their possible inter-relationships without being 
restricted with nodes inter-linked and anchored to one together. 

At no time are students presented with instructions to suggest when and in what order to 
move and insert links between the claim and premises. Students download the jMAP 
application to diagram the supporting arguments consisting of 15 nodes. Once students 
complete the diagram, the students downloaded a second jMAP file to diagram the opposing 
arguments consisting of 14 nodes. Students are instructed to spend no more than 45 minutes 
working on each argument diagram. The mean number of minutes students worked on each 
diagram was 36 minutes (STD = 27, Max = 103, Min = 5). Once the diagrams are completed, 
students submit their diagrams to receive 20 participation points for completing the activity. 
The student receiving the highest score on the diagram for the supporting arguments and/or 
opposing arguments received 10 bonus participation points. The participation points students 
accumulated across all activities in the course comprised of 25% of the course grade. 


Data Analysis 

Scoring the diagrams. One student did not submit the second diagram, so the researcher 
omitted the one submitted by the student to control for practice effect. The 32 diagrams 
submitted from the remaining 16 students were imported into jMAP containing the criterion 
diagram created by the course instructor (Figure 3) to score each diagram across the 
following criteria: a) Percentage of links within the student’s diagram that match those in the 
instructor’s map x 100 (as opposed to the number of links, which students can use to inflate 
their score by merely linking all nodes to one another); b) the number of nodes correctly 
identified as a root premise (a node with only outward pointing arrows and no incoming 
arrows) x 10; c) the number of 1* order premises correctly linked from each correctly 
identified root premise x 10 (A>B); d) number of 2" order premises correctly linked from 
each 1‘ order premise x 10 (ABC); and e) number of 3" order premises correctly linked 
from each 2™ order premise x 10 (ABCD). No points were awarded by jMAP for 2™ 
and 3 order links if any of the downstream links are missing. This criteria measures 
students’ depth of analysis, as used in prior studies (Kim, 2015; Lee, 2012, Shin, 2016). The 
maximum possible score was 280 points. 
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by DIRECTION Name: Instructor Constrasting Map: Student 1 


: re 
Visual Map Comparison Tools Swap fers See) ResetList Help Shared links that match in DIRECTION # of root causes correctly identified (out of 6) 
View my matrix [_etri-t Shared links that don't match in DIRECTION -2_| # of root causes INCORRECTLY identified 
View selected matrix (ctrt-Z to view map) ctri-y ctri-u | Number of links in USER's map # correct 1st order chains from root causes (3 wrong) 
View & compare map to other maps 41 2 % of shared links within User's map (6 of 16) # correct 2nd order chains from root causes (1 wrong) 
ctri-g & ctrlt scrolls through map index 


ctrl-h T ctri-j Number of links in TARGET map # correct 3rd order chains from root causes (0 wrong) 
Number of maps listed in Mapindex = 25 % of shared links within TARGET map (6 of 14) Root cause analysis score 


Note: Green/gray arrows = links present/missing in student x’s diagram; Green halos = correctly identified root 
premises; bottom left are navigation tools to select which diagrams to compare with criterion map; bottom right 
displays diagram scores across scoring criteria; bottom row are buttons used to add links to their diagrams. 


Figure 3. Visual and quantitative comparison of instructor to student’s diagram in JMAP. 


Coding and counting type of process with algorithms. The data from the log files 
recorded 27 mapping actions (e.g., move node, insert link, delete a link, re-route head or tail 
of link to a different node, change density or direction of link) performed by each student 
(including each node’s coordinates before and after each node movement) and were used to 
test various algorithms for measuring the number of times each of the four processes were 
used when placing each node in the diagram. Each log also contained an adjacency matrix 
that stores information on which nodes are linked to one another (Figure 4). With no prior 
instructions on how and where to move the nodes and when to manually insert links to 
connect related nodes, the students constructed the diagrams using four different formats - 
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top-down hierarchy (n = 8), left-to-right structure with claim position near right edge of the 
screen (n = 10), a right-to-left structure (7 = 7), and network/spoke structure (n = 7). Eight of 
the students used the same format to produce both of their diagrams. The other eight used 
one format to produce one diagram and then used another format to produce their second 
diagram. A computer algorithm was used to identify each diagram’s format based on: 1) 
Where the claim node is positioned within three node lengths or width from the top, left, or 
right edge of the screen; and 2) whether the claim node is at least the third closest node to the 
corresponding edge of the screen. This algorithm was iteratively revised until the algorithm’s 
classifications corresponded 100% with the manual classification of all 32 diagrams. 

Using Microsoft Excel’s VBA, two algorithms were formulated (post-hoc versus 
directional) and tested to find the one that can generate backward/forward and breadth/depth 
ratio scores to produce a best-fit regression model for predicting diagram scores. The post- 
hoc algorithm crawls through the log of one completed diagram to determine the 
chronological order in which the nodes are moved for the first time (e.g., NI N5 N7 N10). To 
count the number of times a student performs a backward and forward process, the algorithm 
combs through all the linked nodes in the final diagram recorded in its adjacency matrix. 
Upon finding each pair of linked nodes (e.g., N7 >N1 claim) in the matrix, the algorithm 
checks to see if N7 immediately follows N1 in the node-move-sequence or vice versa. If the 
N7 follows N1, the N7 move is scored as a backward process. If N7 precedes N1, the N7 
move is scored as a forward process. To assign a breadth or depth-first process to a moved 
node B, the algorithm identifies the previously moved node A from the node-move-sequence. 
If B appears on A’s row (indicating AB) or A appears in B’s row (indicating B> A), B’s 
move is scored as depth-first. The B move is scored as breadth-first if the column of B’s 
parent node in the matrix contains the previously moved and sibling node A. 

The other algorithm tested in this study, the directional algorithm, is simpler in form and 
execution. Based on each diagram’s identified format (top-down, left-to-right, right-to-left), 
this algorithm uses the specific move and placement of a node relative to the position of the 
most recently moved node to identify a backward, forward, breadth-first, depth-first process. 
For example, moving premise A into position followed by positioning the claim above 
(midpoint of node A at >= -45° and < +45°) premise A in a top-down diagram indicates 
forward processing. Conversely, moving the claim into a position followed by moving 
premise A directly below the claim in the top-down diagram counts as backward processing. 
Moving premise B to the immediate right of premise A (at >= +45° and < +135°) and below 
the claim indicates breadth-first processing. If premise B were moved below premise A 
(instead of the immediate right or left of premise A), this action indicates depth-first 
processing. This algorithm is also tested using different minimum required distances (from 2 
to 11 node lengths) between a moved node and previously moved node (Taricani & Clariana, 
2006). Because network diagrams are essentially composed of individual segments that are 
top-down, left-to-right, and right-to-left in format (making any directional analysis 
problematic), data from the seven network diagrams were not used to test the algorithms. 


Testing Algorithms with Regression Analysis. Which segment of a log file used to 
predict scores can affect the accuracy of the algorithms because the mean number of times 
nodes were moved to complete each diagram was 80.6 (STD = 63.4, Max = 257, Min =17, N 
= 25), averaging five or more moves per node. Students moved nodes multiple times to make 
structural changes that correct for errors, and to perform cosmetic changes. For example, the 
sequential analysis (Jeong, 2005) of all diagramming actions examined collectively produced 
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the state diagram (Figure 5) that reveals that 86% vs. 11% of actions immediately following 
the placement of a node is the placement of another node versus linking two nodes. Inserting 
a link most often triggers a series of ensuing actions involving relinking, deleting, and 
commenting on the links, which then triggers more movement of nodes. 

To determine if B/F and BR/D ratios are more accurate using only the initial movements 
of nodes, the algorithms generated ratios using all node movements versus the initial 
movements of the first 15 versus 10 versus 5 nodes. For each algorithm, results from the 
regression analysis on the ratio scores generated with each of these four sets of data 
determine which data set produces the best-fit model for predicting scores. For example, the 
initial movements of the first 15 nodes performed in the construction of each diagram are 
used to generate the B/F and BR/D ratios. These two ratios and the associated diagram scores 
for each of the 25 diagrams are entered into the regression model Diagram Score = Bo + 
B,*(B/F) + B,*(BR/D). A one-tailed p-value at .10 is used to test the eight regression models 
because prior studies report the positive impact of backward and breadth-first processing on 
diagram scores and because the algorithms developed in this study are exploratory. 


List of Premises SRBSRBSRLSSSEESSLSES 


media will 


ig Of linguistic 


Note: The - indicates a negative relationship with parent node. First digit conveys impact on scale of 1 = low to 
3 = high. The second digit conveys level of contingency on a scale of 1 = low to 3 = high. 


Figure 4. A matrix that records how each node is linked to other nodes in a diagram. 


151 


ADD LINK 
\ 345>339 


2 a 2 
02 -_ 
$2 8 
8 Aue RELINK 
9 | EFFECT 
1 | 
1611>1598 \ | \ 18>16 
a7 a3 
@ 
, ‘ 06 
f WDIASEIE | ATTRIBUTE 
BUTS ) 137>135 
tr 0s 


18 


Note: Dark arrows identify transitional probabilities between two actions that are significantly greater than 
expected based on z-score tests at p < .05 (Jeong, 2005). The first and second numerical values in each node 
identify the number of times the action was executed and the number of events that followed the action. The 
size of the glow around each node conveys the relative number of times the action was performed. 


Figure 5. Action sequences observed in the diagramming actions across all students. 


Results and Discussion 


Research question 1 - Comparing the algorithms 

The findings in Table 1 show that the directional algorithm using only the first 5 moves 
produced the best-fit model. The post-hoc and directional algorithms were run to compile the 
frequencies of backward, forward, breadth, and depth-first processes across all node 
movements and only the initial movements of the first 15, 10, and 5 nodes. Each set of data 
was then put into separate regression models to produce the p-values, R2, beta weights for 
B/F, and the beta weights for BR/D (Table 1). With the post-hoc method, the mean B/F ratio 
was .596 (STD = .26), and the mean BR/D ratio was .228 (STD = .19). With the directional 
algorithm, the mean B/F ratio was .427 (STD = .42) and the mean BR/D ratio was .679 (STD 
= .28). The directional algorithm using only the first five node movements with no minimum 
required distance from the previously moved node (all varying limits on node distances 
severely suppressed backward and forward process counts to produce non-significant 
regression models) produced the B/F and BR/D ratios with the best-fit regression model. 
With the diagram scores ranging from 10 to 97.8 (M = 39.4, STD = 22.43, n= 25), a 
significant regression equation was found (F(2, 22) = 3.01, p = .069), with a R2 of .215. 


Table 1 
Results from the Regression Analysis across All Algorithms 
First Five First Ten First Fifteen All Moves 
Direct Posthoc Direct Posthoc Direct Posthoc Direct Posthoc 
p O69 280 367 730 282 TBO bs) P30 
Re 215 109 OBE O28 108 22 O02 28 
E/F 20.68 25.14 4.81 -6.48 13.24 -3 71 1.61 -9.99 


BRD  -24.62 -26.83  -36.63 5.93 “50.53 5.96 8.05 13.52 


* Significant at they < 10 level. 
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A comparison of the two algorithms (post-hoc vs. directional) and their regression 
models reveals that node placements at the beginning of the diagramming process provide 
the best indication of what processes students are using to analyze arguments. In contrast, 
movements occurring later in the diagramming process provide poor indicators of students’ 
reasoning process. The explanation for this finding is that students move many of the nodes 
to their desired location before moving the nodes again (each node was moved an average of 
5 or more times) simply to space them properly to make room to insert node-linking arrows 
into their diagrams, re-positioning nodes to revise and correct for errors in the diagram, and 
making other cosmetic changes (as explained earlier). These behaviors also explain why the 
directional algorithm performed better than the post-hoc algorithm (where B/D and BR/D 
ratios are determined by crawling through each pair of linked nodes in the completed final 
diagram and determining which of the paired nodes was moved after the placement of the 
other node). For example, Figure 5 shows that the students moved or re-positioned a node 
immediately after they re-linked the tail of an arrow to a different node (Relink Cause) at 
52% of the time, re-linked the head of an arrow to a different node (Relink Effect) at 69% of 
the time, and after deleting a link (Delete Link) at 32% of the time. Moving nodes in the 
process of making revisions are isolated events that don’t follow a sequence that conforms to 
or reflect the systematic use of backward and depth-first processing. 

Overall, the findings suggest that diagramming tools can identify students’ reasoning 
processes by simply observing the first five node placements. The other alternative is to 
analyze the first five moves immediately following the first time placement of the claim node 
(which was placed during the first or second move on average). A regression model revealed 
that the chronological order in which the claim node is placed approached statistical 
significance at p = .110, (F(3, 21) = 2.59, p = .079), with an R2 of .270. The later the claim 
node is placed, the lower the score. This suggests that students can create better diagrams if 
the claim node is placed at a default location from the start so that the directional algorithm 
can immediately implemented after the fifth node placement to assess a student’s reasoning 
process. In contrast, the post-hoc method produces not only inaccurate assessments of 
students’ reasoning processes, it can only assess students’ reasoning processes (and deliver 
corresponding feedback) after the completion of the diagram. 


Research question 2 — Breadth-first and backward processing 

The best fit model using the B/F and BR/D ratio scores produced with the directional 
algorithm using only the first five moves revealed that greater use of backward processing 
over forward processing and the greater use of depth-first processing over breadth-first 
processing was associated with higher diagram scores. The best-fit model was diagram score 
= 46.4 + 20.6*(B/F) -24.6*BR/D. Both B/F and BR/D were significant predictors of diagram 
scores, with p = .038 and p = .059, respectively. The findings from the best-fit regression 
model show that a higher B/D ratio (or increasing the use of backward processing) increases 
scores. This finding is consistent with prior categorical findings on the use of backward 
processing to construct causal diagrams (Lee, 2012; Shin, 2016) and argument diagrams 
(Kim, 2016). This finding also validates the argument diagramming tool, REASON 
(ThinkReliability, 2007), that explicitly prescribes the use of backward processing. However, 
this study found that increases in the ratio of BR/D processing were associated with 
decreases in scores, which suggests that increasing the use of depth-first processing increases 
scores. This finding is not consistent with prior studies that found some but limited evidence 
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that breadth-first processes produced more accurate diagrams (Kim, 2016) and causal 
diagrams (Shin, 2016) than depth-first processing. In these prior studies and in the current 
student, all nodes were specified in advance on screen and were not generated by students. 

One of the four possible explanations as to why breadth-first had a negative (not positive) 
effect on scores is that the algorithms do not account for other behaviors associated with 
breadth-first processing. Another possible explanation is that the arguments analyzed in this 
study may be more complex than arguments analyzed in prior studies given that the scores in 
this study were very low in relation to the maximum possible score. To compensate for the 
increased difficulty, students may be more likely to use the depth-first process because depth- 
first processing places lesser demands on memory than breadth-first (Al-Ajlan, 2015). 
Furthermore, students are more likely to use depth-first processing when a clear outcome or 
conclusion is given from the start (Sharma, 2012). In this study, a clear claim or conclusion 
was presented to the students. The correlation between the sum frequency of backward + 
forward processing and frequency of depth-first processing was significant, 7(23) = .788, p = 
.001. No significant correlations were found between any of the four individual processes. 

A fourth explanation is that the use of breadth versus depth-first may not be a zero-sum 
game and that there may be times when performance can improve by increasing the use of 
both processes. However, the regression analysis using the observed frequencies of 
backward, forward, breadth-first, and depth-first (as four independent variables) was not 
significant (F(4,20) = 1.33, p = .291). Also, the regression analysis using the sum of 
backward + forward and sum of breadth + depth-first processes also was not significant 
(F(2,22) = 1.40, p = .266). One more possible explanation is that depth-first may be more 
effective when used to identify root premises and the 1‘, 2", and 3" order nodes linked 
upstream from root premises - a measure that was not used in prior studies that found 
breadth-first processing to be more effective. The correlational data suggests that this might 
be the case because the regression analysis of scores without the above scoring criteria was 
not significant (F(2,22) = 1.34, p = .281, R2 = .109) but was significant with this criteria 
(F(2, 22) = 3.01, p = .069, R2 = .215). However, no cause-effect relationships can be 
determined at this time from these correlational findings. 


Research question 3 — Relative impact of backward vs. breadth-first processing 

The best-fit model with diagram score = 46.4 + 20.6*(B/F) -24.6*BR/D shows the use of 
backward and breadth-first processing have nearly equal but opposite impact on diagram 
scores. Inversely, this finding suggests that the use of backward processing and depth-first 
processing has a nearly equivalent impact on scores and that both processes can play a 
necessary role in diagramming complex arguments. However, the inherent structure of the 
given argument (from few to many branches, from short to long branches) and the criterion 
used to assess the quality of an argument diagram could determine whether using one process 
has a greater impact than using the other. Furthermore, students’ expectations on what 
hierarchical structures might or might not exist in arguments can also determine which 
process they use to complete a diagram. Another possible explanation is that students may 
choose to use a specific process to perform a specific step in the diagramming task (scan 
claims, identify the goal, move nodes, link nodes, review and revise links) — particularly 
steps that are used to create good diagrams (Aguiar & Correia, 2017). For example, breadth- 
first processing may be more effective when students attempt to move and sort nodes 
according to their resemblance in the level of generality and to simplify the problem space 
(Kim, 2015). This behavior is similar to the way high performers sort and group pieces of a 
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jigsaw puzzle based on their similarities before attaching the puzzle pieces (Antle, 2013). 


Implications for Designing Diagramming Tools 

The findings in this study suggest that algorithms can be used in diagramming tools to 
identify, assess, and scaffold the use of backward processing and depth-first processing to 
help students analyze complex arguments with greater accuracy and efficiency. Overall, 
designing diagramming tools to facilitate the use of backward and depth-first processing can 
be given equal emphasis because the use of both processes was found in this study to have an 
equal impact on diagram scores. To help students use both of these processes more 
effectively, diagramming tools should prompt students at the start of the diagramming task to 
identify and place the conclusion first before they are allowed to move and position other 
nodes. By doing so, diagramming tools can identify which map orientation (top-down, 
bottom-up, left-right, right-left) a student is using to construct the map so that the tool can 
identify mapping processes accordingly. Once the conclusion is placed, students are then able 
to use backward and depth-first processing to work from conclusions to root premises. The 
alternative is to create diagramming tools that require students to place the conclusion at a 
default location on-screen so that students use the desired map orientation. 

To create diagramming tools that can identify the processes students are using, 
diagramming tools can record where students position the first five nodes that students move. 
The tools can then determine which process is used by noting where each node is spatially 
placed in relation to the most recently moved node. Once the processes used to move the first 
five nodes are identified, diagramming tools can identify each students’ primary or preferred 
mapping process by computing the ratio of backward to forward processes and the ratio of 
breadth- to depth-first processes performed by each student because: 1) Students tended to 
use all four processes to some extent during the first five moves; and 2) the simple frequency 
counts of each process alone (forward, backward, depth-first, breadth-first) did not predict 
students’ diagram scores. Diagramming tools may be able to identify mapping processes with 
greater accuracy if the tool automatically and by default inserts link(s) whenever students 
place a node within a certain area and distance from another node or between two nodes. 

To help students use more backward processing, re-design diagramming tools so that two 
nodes can be linked together only by moving and positioning a node directly below another 
node. To prescribe greater use of depth-first processing to help students better identify root 
premises and chain together premises that link root premises to main claims, diagramming 
tools can prompt students to identify the next node down the chain immediately after linking 
two nodes. To achieve the benefits of using depth-first processing, diagramming tools must 
apply criteria (like those used in this study) that measures the depth of analysis and not just 
the number of correctly identified links. These types of constraints and system prompts can 
be faded out gradually as students master the mapping skills. 


Directions for Future Research 

In summary, this study used a constraint-free diagramming tool (integrated with data 
mining) to explore possible approaches to operationalizing the measurement and 
identification of strategies used to predict and better analyze arguments. The baseline 
findings and the noted limitations provide a framework for conducting future research. The 
methods developed in this study will hopefully: 1) open the door to new lines of research to 
achieve a more fine-grained understanding of the underlying processes behind argument 
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analysis; 2) provide future guidance on how to develop diagramming tools that can verify 
and gauge students’ use of target processes, provide real-time performance feedback, and 
provide opportunities for more extensive and more efficient practice; and 3) help us to 
identify how to design diagramming tools that can guide students through the process of 
analyzing and achieving a deeper understanding of domain knowledge embedded in complex 
real-world problems. 


Future research can build on this study’s findings by: 1) validating the mined data and 
algorithms by analyzing video and think-aloud protocols while using diagramming tools that 
can playback the log data to closely examine the diagram construction process; 2) using 
larger sample sizes to test for cause-effect relationships between process and outcome; 3) 
comparing the effects of diagram format; 4) comparing the use of B/F and BR/D processes 
between more versus less complex arguments; 5) determining what effects specific 
diagramming actions (e.g., automated versus manual linking) have on the use of B/F and 
BR/D processes; and 6) creating diagramming tools that can measure and control for 
individual differences in mapping skills so that the tools can be used to conduct large-scale 
assessment of students’ understanding of domain content independent of their mastery of 
mapping skills. 
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Introduction 


The purpose of this study was to examine whether the use of real-world context and two 
types of feedback with e-learning affected learning outcomes and motivation over time for 
learners in medical laboratory science (MLS) and medical laboratory technology (MLT) 
programs. This study examined incorporating the scenario of landing a dream job in a medical 
lab, performing virtual lab tests (manual sperm cell counts) and reporting the results. The study 
also incorporated feedback related to the reported results. Because MLS and MLT professionals 
will conduct laboratory tests and report those results to physicians, it was thought that delivering 
contextualized, online instruction with ‘virtual’ consequences, manual cell count instruction 
could be less abstract and more meaningful to learners, which might help improve both 
motivation and learning outcomes. 


Literature Review 


As far back as 1989, Brown, Collins, and Duguid acknowledged that, “Authentic 
activity... is important for learners, because it is the only way they gain access to the standpoint 
that enables practitioners to act meaningfully and purposefully” (p. 36). The authors suggested 
that in traditional education settings, students may be able to pass exams, but not be able to use 
the conceptual knowledge in practice. To address the disconnect between knowing and doing, 
they recommended the use of cognitive apprenticeships, or situated learning, to contextualize 
learning using the social and physical context of the instruction. In 1995, Choi and Hannafin 
suggested that situated learning anchors knowledge and skills in realistic contexts. 

Scenario-based learning, and later scenario-based e-learning, grew out of this idea that 
situated learning provides context for how information will be used in the real-world (Errington, 
2011). According to Errington (2010), scenario-based learning refers to educational approaches 
that use scenarios to bring about desired learning outcomes. 

Clark and Mayer (2011) asserted that scenario-based learning goes by many names, such 
as whole-task, case-based, problem-based, and immersive instruction. Researchers have 
examined the use of scenarios as preparation for flipped learning in which lecture-based material 
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is delivered online and then applied in person (Lehmann, Bosse, Simon, Nikendei, & 
Huwendiek, 2013). 

Errington (2010) asserted that scenarios have the potential to provide rich practical 
experience that extends past conventional lectures and tutorials and Clark (2016) suggested that 
“Scenario-based learning can lead to better transfer to tasks different from those used in the 
training and to greater long-term retention” (p. 55) and that it should be considered “in situations 
in which on-the-job experience is rare, dangerous, or impractical” (Clark, 2013, p. 13), but she 
cautioned that more research is needed before we can extend these conclusions to all situations. 

Feedback is tied closely to all types of instruction and Keller (2010) suggested that 
feedback should be provided to help improve student confidence and show learners how to take 
corrective action. Similar to scenario-based learning, feedback types go by different names. In 
general, corrective feedback indicates only the correctness of a learner’s response; whereas, more 
elaborative or explanatory feedback not only provides the correctness of a learner’s response but 
also includes additional information about why an answer was incorrect and how to rectify the 
mistake. 

Based on the results of their analysis of 12 meta-analyses in 2011, Hattie and Gan 
asserted that feedback is a powerful influence on learning outcomes. Feedback has also been 
found to play an important role in computer-based instruction. In 1995, Azevedo and Bernard 
extracted effect sizes from 14 studies examining feedback and suggested that feedback has to be 
regarded as one of the most critical components of computer-based instruction. 

Keller (2010), Hattie and Gan (2011) and others have also suggested that feedback can 
impact learner confidence and motivation. However, despite there being a multitude of research 
on feedback, the findings are varied and inconsistent (Shute, 2008). Lechermeier and Fassnacht 
(2018) suggested that “Although an extensive body of research has stressed its importance, a 
conclusive overall picture on feedback characteristics effects is missing” (p. 145). 

Similar to feedback, although a tremendous amount of research has been conducted on 
motivation, there is little guidance for those who want to be more effective at motivating learners 
(Keller, 2010). Therefore, Keller developed his ARCS model of motivational design for learning, 
which includes attention, relevance, confidence, and satisfaction components. 

Attention refers to capturing and sustaining learner attention, relevance refers to the 
perceived usefulness and importance of the instruction to the learner, confidence refers to how 
confident the learners are they can be successful, and satisfaction has to do with learner 
expectations and how they feel about their performance. According to Keller (2010), these 
elements can affect how much effort learners put into accomplishing a goal. 


Methods 


Two research questions were examined for this study. The first research question 
involved whether scenario level (present or not present) and feedback type (intrinsic or 
elaborative) had significant main effects over time on college students’ ability to manually count 
sperm cells or whether there was an interaction between these two independent variables on 
learning outcomes as measured by the researcher-developed knowledge instrument. 

The second research question was whether scenario level and feedback type had an effect 
on learners’ motivation over time or whether there was an interaction between the two 
independent variables on motivation as measured by Keller’s (2010) Instructional Materials 
Motivation Survey (IMMS). The IMMS is a 36-item self-report measure of students’ attention, 
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relevance, confidence, and satisfaction. Its validity and reliability have been described by Keller. 
The knowledge instrument was designed to measure whether students could recall the procedure 
of counting sperm cells and doing the necessary computations to report an accurate total cell 
count. Content validity was established using the Delphi method with experts from the content 
area. 

As such, this study made use of a repeated measures design with two between groups 
factors, scenarios (present or absent) and feedback (elaborative or intrinsic). Both instruments 
were administered at three time points with the survey being administered first in case learners’ 
content performance might have affected their motivation. 


Treatments 


All of the instructional treatments were designed as web-based modules using the 
existing text-based standard operating procedure and the expertise of an MLS Body Fluids and 
Urinalysis course as well as other MLS professionals. Each treatment consisted of a worked 
example followed by two practice activities for learners to complete and those practice activities 
included either intrinsic or elaborative feedback. All four treatments contained the same 
information, but it was presented differently based on the scenario and feedback levels. 

The scenario-based treatments started with, “Congratulations! You've just landed your 
first dream job in a medical laboratory. One of the first tests that have come in is a semen 
analysis...” Then the lesson led learners into the patient scenario and what they would be doing 
as part of their new job. The remaining lesson pages were also tied to performing job duties as 
guided by their new supervisor, Dave. 

The two treatments without scenarios began, “This lesson is designed to show you how to 
manually count sperm cells using a standard hemocytometer. A hemocytometer is a specialized 
microscope slide.” The remaining lesson pages contained the same information as the scenario 
treatment, but presented the information directly without tying it to patients or working in a 
medical lab. 

The intrinsic feedback provided learners with consequences for their responses. So not 
just informing participants whether they were correct or incorrect, but also the impact of their 
reported counts. For example, the intrinsic feedback stated, “Although it was within the reference 
range, your count was significantly different from Dave's calculation of eight hundred fifty-seven 
million 500 thousand. Such an erroneous calculation could have negative clinical consequences.” 

Elaborative feedback on the other hand informed learners whether they were correct or 
incorrect and provided them guidance to rectify incorrect answers. For example, the elaborative 
feedback for the same question stated, “Incorrect, the correct answer is eight hundred fifty-seven 
million 500 thousand. It looks like you may have calculated incorrectly.” 

Elaborative feedback was presented without any context of the medical lab. Both 
feedback types included a summary of the correct steps as well as the values that should have 
been obtained. 

Because the main interest in this study was the efficacy of a scenario-based instructional 
approach with intrinsic feedback, the no scenario with elaborative feedback treatment served as 
the control group for this study. 


Data Collection 
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The target population for this study was MLS and MLT students. This was done to be 
able to provide real-world, job-related scenarios and relevant feedback. Although faculty in such 
programs at other institutions across the United States expressed interest in being involved in this 
study, no students from those programs participated; therefore, the majority of participants were 
from the researchers’ institution and also included participants from other programs at the 
institution. 

Due to the limited number of participants, the study materials were delivered four times 
in an effort to increase power for inferential statistics. During each iteration, data were collected 
over seven weeks. Participants were randomly assigned to one of the four treatment groups that 
were differentiated by the two independent variables. During the first week, informed consent 
was obtained and participants submitted a demographic survey, which was collected 
anonymously strictly for describing the sample. After that, participants completed the pre-IMMS 
followed by the researcher-developed knowledge pretest. During the fourth week, the 
instructional interventions were delivered via the learning management system followed by the 
immediate-post IMMS and then the immediate knowledge posttest. Lastly, participants 
completed the delayed-post IMMS and delayed knowledge posttest during week seven. After all 
participants had completed the study materials, the data were exported from the learning 
management system to Excel, where they were anonymized, and coded for analysis using IBM 
SPSS. 


Data Analysis 


Descriptive statistics were used to provide a complete description of the sample using 
participants’ reported demographic information. For research question one, a 2x2 repeated 
measures analysis of variance, ANOVA, was used to determine whether any main effects for 
scenarios or feedback existed or whether there was an interaction between those two independent 
variables on learning outcomes over time. A second 2x2 repeated measures ANOVA was used 
for research question two to determine whether any main effects for scenarios or feedback 
existed or whether there was an interaction between these two independent variables on 
motivation over time. Additionally, Chi-square tests of independence were used to determine 
whether there was a relationship between participants’ treatment groups and their ability to 
manually count sperm cells. 


Results 


The intended sample was MLS and MLT students so the real-world, job-related scenarios 
and intrinsic feedback would be relevant to their chosen profession. Therefore, attempts were 
made to solicit participants from outside institutions; however, although faculty at 10 outside 
institutions expressed interest, no learners from those institutions participated, which had a major 
impact on the number of participants obtained as well as trying to ensure relevance of the 
scenarios and intrinsic feedback to these majors. 

Since a limited number of participants from MLS and MLT programs were obtained, 
university students from other programs within and outside of ISU were invited to participate. 
These additional calls for participants resulted in the study being conducted a total of four times 
(N = 84). 
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The first iteration of the study included 48 undergraduate and graduate students taking a 
Urinalysis and Body Fluids class as part of their Medical Laboratory Science program at a 
medium-sized university in the West during the fall 2019 semester (NV = 48). The subsequent 
iterations included the students from other programs from within the same university as well as 
outside the institution (NV = 36). Each iteration saw fewer numbers and by spring of 2020, many 
schools and other services shut down due to COVID-19. 

The descriptive statistics showed that 60% of the participants were non-traditional 
students, as defined by being 25 or older, 63% were female, and 60% were enrolled in a MLS 
program. Data were included for a total of 84 participants in the sample descriptives because 
there was no way to de-anonymize the demographic survey data. 

The majority of learners (92%) had previously taken an online course and had taken an 
online exam or assessment (94%). Nearly 70% of the participants had used the same learning 
management system that was used to deliver all of the study materials and 81% had not had 
previous instruction on the topic covered in the instructional treatments, manual cell counts. 

For research question one, participants’ content acquisition and cell count accuracy, the 
data were analyzed with and without three outliers identified in SPSS and both analyses yielded 
similar results. No statistically significant effects were observed for scenario level (present or 
not) or feedback type (elaborative or intrinsic), nor was an interaction observed for these two 
independent variables; however, there was one statistically significant result, which was for time. 

Pairwise comparisons using a Bonferroni adjustment saw a statistically significant 
increase from the pretest to the immediate post test, which would be expected as a result of the 
instruction; however, a statistically significant increase was also observed from the immediate 
posttest to the delayed posttest, which was somewhat unexpected because of the two-week delay 
between those two posttests. 

A Chi square analysis was used to examine participants’ mastery for performing manual 
cell counts accurately. Correct answers were defined as within plus or minus one standard 
deviation of the mean of experts' counts. Mastery was defined by answering at least two out of 
three cell count questions correctly; whereas, not having met mastery was defined by answering 
one or fewer cell count questions correctly. The results of the Chi square analysis showed no 
statistically significant differences for the immediate or delayed posttests and the treatment 
conditions did not appear to have an effect on participants' accuracy of manually counting sperm 
cells. 

In contrast to the results on the content knowledge scores, participants’ skill mastery 
dropped between the immediate posttest and the delayed posttest, which would be expected after 
the two-week delay. Although not statistically significant, higher mastery levels were observed 
when intrinsic feedback was paired with scenarios and when elaborative feedback was paired 
with no scenarios on both the immediate and delayed posttests. 

For research question two, examining the effect of scenarios and feedback type on 
participants’ motivation over time, the results of the 2x2 repeated measures ANOVA indicated 
there was not a statistically significant difference in IMMS scores over time based on scenario 
level or feedback type. Additionally, no statistically significant interaction between the two 
variables was observed. 

Similar to the knowledge scores, the only statistically significant main effect observed 
was for time. Pairwise comparisons showed a statistically significant decrease in motivation 
from the pre-IMMS to the immediate post-IMMS. Although not statistically significant, a slight 
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increase from the immediate post-IMMS to the delayed post-IMMS was observed for all 
treatment groups except those who received elaborative feedback with scenarios. 


Interpretations 


Although no statistically significant differences were observed for research question one, 
the line graphs in Figure 1 show an apparent interaction effect. For the scenario groups, 
participants who had scenarios with intrinsic feedback had a lower starting score, but caught up 
to the participants who had elaborative feedback on the immediate posttest, and then surpassed 
them on the delayed posttest. 


Figure 1 
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As shown in Figure 2, participants who received intrinsic and elaborative feedback with 
scenarios had similar means on the pretest and immediate posttest, but those who had elaborative 


feedback appeared to score higher on the delayed posttest than participants who had intrinsic 
feedback with the scenarios. 
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Figure 2 
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Looking at the results of this study as more of a pilot test due to the low number of 
participants, it appears the learners in all treatment groups increased their knowledge of manual 
sperm cell counts regardless of the scenario level or feedback type from pretest to immediate 
posttest and again from the immediate posttest to the delayed posttest. It is possible this increase 
was due to the immediate posttest acting as a subsequent learning opportunity. Further, the 
participants appeared to have learned recall, but not application, which actually decreased from 
the immediate to the delayed posttest. This could indicate that without sufficient skill’ practice, 
the skill was not necessarily learned (Fisher et al., 2018). 

For research question two, participants’ IMMS ratings decreased significantly from the 
pre-IMMS to the immediate post-IMMS, which could have been due to participants being more 
interested in the study at the start, but having lost interest after waiting two weeks or after having 
gone through the instructional intervention. 

Although not statistically significant, a pattern also emerged for the IMMS results. As 
shown in Figure 3, while participants in both scenario groups had lower IMMS scores from the 
pretest to the immediate posttest, participants who had intrinsic feedback rated their motivation 
slightly higher on the delayed post-IMMS than those who had elaborative feedback paired with 
scenarios. 
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Figure 3 
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Conclusions/Recommendations 


Because the lack of statistically significant findings could have been due to the small 
sample size the first recommendation is for future research using a larger sample. Although the 
patterns that seemed to emerge were not statistically significant, if future research supports those 
patterns, a possible recommendation would be to pair intrinsic feedback with scenarios and 
elaborative feedback without scenarios for optimal knowledge acquisition and improved 
motivation. Future research is recommended to confirm any possible significance of these 
patterns with a larger sample and with different disciplines. 

Although knowledge scores increased over time, mastery levels did not. Even though that 
would be expected due to the delayed time after the instruction, it might also indicate that the 
instructional treatments were not effective for teaching skill acquisition. Therefore, future 
research examining the effectiveness of scenario-based e-learning for teaching conceptual 
knowledge versus skill acquisition is recommended. 

Further, acquiring skills requires practice so it is possible that the instructional treatments 
did not include enough practice opportunities. Clark and Mayer (2011) suggested that evidence 
supports spacing practice over time and future research could help determine whether spaced 
repetition would be beneficial for teaching manual cell counts. 
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Introduction 


A voluntary social year, or what is more known in the United States as a “gap year,” 
typically takes place after the students have finished their high school education and before they 
enroll in higher education, such as a university or a training program. In Germany, Freiwilliges 
Soziales Jahr (FSJ), voluntary social year, is “a period of time, generally from six to 18 months, 
spent doing voluntary work ... [d]Jesigned to offer young adults the chance to find their 
vocational or academic orientation while contributing to the greater social good...." 
(Volunteering in Germany, 2018, para. 3) During the social year, students may stay living at 
home, work at a different city in Germany, or even work and live in a different countries, such as 
Brazil. In their placements, students typically work with a community that is in need of help, 
such as a childcare facility, a school, or a hospital. 

Although the voluntary social year is quite popular in Germany and even throughout the 
rest of Europe, the United States seems to view the extended time taken between the end of high 
school and secondary education as something that is uncommon and not often used. While 
students in Europe use this time to volunteer, learn new skills, and/or help others, many 
American students use this time to join the work force and figure out what they feel is best for 
them while they are out of high school. Americans view this as more of a period of personal 
growth and reflection, rather than an avenue in which one may further their education through 
exposure to other cultures and volunteerism in a social area. 

This paper focuses on “Voluntary Social Year” (Freiwilliges Soziales Jahr) in Germany. 
Investigating the issue from the perspectives of German high school graduates and their parents 
is of interest to the authors. From this paper, the readers will gain a deeper understanding of daily 
life of the German high school graduates who have participated in a voluntary social year and be 
informed of how technology has been used by the students during this special period of time of 
their lives. 


Literature Review 
The literature review primarily focused on three aspects: the perception that both students 
and parents have about the idea of a voluntary social year, the use of technology to engage in 


informal learning while participating in the social year, and how the social year affected students 
when they returned home. 
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Perceptions of the Voluntary Social Year 


One of the main purposes of this study is to investigate the students’ and parents’ 
perceptions of the voluntary social year. This can pertain to not only the perception that one may 
have before the social year begins but also the perception of how the social year may have 
affected the students after they returned home. 

Although not discussing a typical social year, Clerkin (2019) researched the perceptions 
of a “Transition Year” (TY) that surfaced throughout 20 schools in Ireland in which students 
participated in a similar program during grade 10, which acts as a quasi-gap year, since instead 
of the gap year occurring after secondary school, it is offered midway through secondary school, 
for example, between grades 10 and 11. Although this is done in a different time frame than the 
voluntary social year that is being discussed, the activities and assignments are similar to those 
that are being discussed. Discussions and conclusions within Clerkin’s research determined that 
most students who participated in TY felt more mature in their last 2 years of schooling, whereas 
students who did not participate in TY felt less mature in those last years. In addition, there were 
noticeable differences in the responses from males and females, with females claiming they were 
more independent and self-reliant, whereas the male counterparts were reporting to be less self- 
reliant and more immature when they did not participate in TY. 

Martin (2010) sought to examine the predictors of Australian students’ intentions to take 
a gap year and the academic profile of students who have completed a gap year. Martin 
specifically looked at motivation, demographics, and “postschool uncertainty” to try and 
determine the motivating factors behind the allure of a gap year. His hypothesis stated that 
students with lower academic motivation and those with more uncertainty over postschool plans 
would lead to the intent to take a gap year. The conclusion was made that the gap year can mend 
the lower motivation and in fact encourage a study to achieve more success in university after 
completing a gap year. However, further research is needed to help corroborate or disprove these 
hypotheses due to the assumption that lower academic motivation was the sole contributor to 
students’ interest in participating in a voluntary social year. 

There are few studies that address the students’ perceptions of the social year. What was 
not addressed in these studies was the perceptions that parents may have of the voluntary social 
year, and whether they felt it was beneficial or detrimental to their children. This is a major 
plothole in the previous research of the social year because ultimately, students are more likely to 
seek the approval of their parents before embarking on one of these trips. It is also possible that 
if parents view this as a negative pursuit, students may be more reluctant to take on a social year 
experience. 


Technology and Informal Learning During the Social Year 


Due to students’ lack of formal education during a social year, it is important to take a 
look at what students may be doing to informally further their education and gain knowledge 
during that time. 

Mills, Knezek, and Khaddage (2014) conducted a study in which US undergraduate 
students were given a survey to reveal the attitudes of students and their use of mobile devices to 
obtain information. This research study suggested that a better understanding of technological 
advancements be critical to the shift from formal to informal learning. The authors stated, 
“Cross-validation with an established Mobile Learning Scale ... indicates that Information 
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Sharing aligns significantly ... with Mobile Learning. Information Seeking, Information Sharing, 
and mobile access are presented as important ... in the shift along the formal to informal learning 
continuum.” (p. 324) The authors further stressed the importance of not only knowing what 
students may be informally learning but also how they may be acquiring this information. 

Galanis, Mayol, Alier, and Garcia-Penalvo (2016) suggested a higher emphasis on peer 
interactions and social settings to encourage informal learning. The researchers asked, “Can the 
knowledge obtained through non-formal and informal means be quantified and evaluated in 
order to be formalized and recognized within the competences of a curriculum?” (p. 597). 
Although the authors did not directly answer the question, students within the social year 
program are exposed to different cultures and languages when they travel abroad which 
coincides with the authors’ suggestion to emphasize a more social atmosphere when it comes to 
learning. In theory, the idea of moving to another country and/or city in itself can breed informal 
learning in ways students may have not thought possible. 

Studies may provide information about informal learning and how informal learning can 
be done; but there is very little information about how informal learning is done during a social 
year. These studies also do not address how students may be informally learning new languages 
and cultures by simply living and working in their respective programs. Many students involved 
in a type of social year program go on to attend other training programs or potentially university, 
but previous research lacks in attempting to seek out the truth of how students stay current in 
their education. 


The Impact of the Social Year 


The sign of whether a voluntary social year was beneficial to both the organization being 
served and the individual in the program is to see how they are affected by the experience after 
the fact. When researching this topic, it was noteworthy to pay attention to how the students and 
communities reacted to the ending of the program. 

O’Shea (2011) conducted interviews with students in the UK who returned home from 
their voluntary social year in which he asked these students about their experiences during their 
time abroad and how these experiences have shaped them into the persons they are today. From 
these interviews, he was able to gauge students’ motivation, culture shock, psychosocial 
improvements, and other aspects. He noted that although some students participated in the 
volunteer gap year for completely altruistic reasons, there were many that simply did it so they 
could avoid going to university or use the time to figure out what they truly wanted to pursue in 
life. This is contradictory to what many organizations strive for within their programs with 
students, but is similar to the American idea of a “gap year” in which they use their time to delay 
higher education. However, it was determined that overall students came back from their 
voluntary social year with a heightened sense of self and gained experience and maturity through 
their time abroad. 

Wilde (2016) discusses the dichotomy between wanting to help others while also being 
forced to reflect on how a person’s actions affect others, and concerns herself with the idea that 
volunteerism in foreign countries merely help people experience the culture for a short period of 
time, but does not help them gain accurate knowledge and insight on the problems that these 
countries face. Wilde stated, “What emerged from my fieldwork was an emphasis on 
individually orientated [sic] activities, in contrast to the ethos of the charity that focus on 
interconnections and awareness of an individual’s impact upon others around them” (p. 79). She 
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found that these social year programs focus more heavily on the interconnected web of relaying 
global development information, rather than focusing on individualizing their experiences to 
foster better learning. This could directly affect the idea of “informal learning” that students 
should be experiencing and questions whether the students are getting enough information about 
that environment to consider it educational. 

Although the impact may be questionable to those students it has served, most research 
indicates that social year positively impacts the student by increasing maturity and/or 
independence when speaking about their time during their gap year. King (2010) reported that, 
after interviews with 23 students from Southern England who participated in a social year, the 
interviews illustrated that the gap year “enabled these young people to indicate, in this context, 
that their current self differed from a past self’ (p. 353). This conclusion is important for our 
research because it helps support the fact that students benefit from this time in a gap year also 
by growing and changing as people. 

In this field, there is a void of information of not only the participating students to be 
interviewed, but also the families and/or communities who are supposed to benefit from the help 
that the student provides. However, one can hope that the impact of the social year would be 
beneficial for those who are volunteering and working during that time. Ultimately, what is at 
stake is the impact that the experience has on the students when they return home, including the 
families and communities in which they serve. 


Methodology 


The goal of the research was to gain more knowledge about the process of the social year 
and how the process has affected both parents and students alike. In addition, the authors 
investigated how technology was used by the students during the social year. 


Research Inquiry/Questions 


While investigating this topic, five questions were developed in order to guide the 
research and to use as the focus for this project. They are listed as follows: 

1. What are the perceptions of German high school students about a “Voluntary Social 
Year”? 

2. What do German high school graduates who have participated in a “Voluntary Social 
Year” think of their experiences? 

3. What do parents of German high school graduates who have participated in a “Voluntary 
Social Year” think of their children’s experiences? 

4. What technology was used for communication and learning during the social year? 

5. How was technology used for communication and learning during the social year? 


Data Collection 


Data was collected through surveys and interviews from participants who were a 
convenience sample. This was done in order to collect qualitative data to examine and determine 
how students and parents both perceive the idea of the volunteer social year. 

The surveys were given via Google Forms to high school graduates who completed 
secondary school in the years 2012-2018. These were given in order to not only obtain contact 
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information from the students but also gain a basic understanding of the students’ experiences. 
Additionally, the survey’s purpose was to gauge which students would be open to participating 
in interviews in which they would go into more detail about their participation in the social year. 
Included in the survey were four questions with sub-questions attached to questions two and 
three. Previous information about the program, placement in the program, and technology use 
during the program were among the questions in the survey. The last question inquired as to 
whether the student would like to participate in face-to-face interviews. Below were some 
questions in the survey: 

“2. Were you aware of the ‘Voluntary Social Year’ program while you were deciding on your future, e.g., 

what to study? 

2.1. If yes, how did you learn about it? Did you consider participating in the ‘Voluntary Social Year’ 

program? Why or why not? 

3. Have you participated in the ‘Voluntary Social Year’ program? 

3.1. If yes, where did you go? How long were you there? What did you do? What do you think of the 

experiences? How did/do the experiences help or change you? Knowing what you know now, would you 

still make the same decision? Why? 

3.2. If yes, what technology did you use for communication/learning during the social year? How did you 

use it, e.g., for staying in contact with your family members? 

3.3. Ifno, did or do you wish that you had participated in the 'Voluntary Social Year' program? Why or 

why not? Knowing what you know now, would you still make the same decision? Why?” 

A total of 23 students responded to the survey. It was planned that students completed the 
survey in early June 2019, prior to the interviews. However, many students completed the survey 
late, after the interviews were conducted. 

Interviews were conducted with both parents and students: four parents and fourteen 
students were interviewed. Interviews were conducted in late June of 2019, with additional 
interviews being conducted during Fall of 2019. In June, interviews were done with two focus 
groups and one individual. The focus groups consisted of students who were in medicine, teacher 
education, and computers and engineering. Ten open-ended questions were given to students 
about their experiences and how they felt this process had influenced them. One set of parents 
was interviewed about their experience of having their children complete the program. These 
questions were similar in content to the interviews done with the students but was more focused 
on how they were affected, what they saw in their children when they returned, etc. During Fall 
of 2019, another set of parents and a student were interviewed using the same interview 
questions. All interviews were recorded using a recording device. 


Data Analysis 

Qualitative data obtained from the survey and the interviews were analyzed using both 
coding and content analysis to determine any similarities or trends between the responses of the 
individuals. Analysis was conducted separately by two people in which they listened to the 
recordings of the interviews and made notes and conclusions in order to cross-check the analysis 
results. 


Results 


Twenty three students completed the survey. Among them, 14 students participated in a 
social year while nine students did not participate. Students who did not participate in a social 
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year were asked, “did or do you wish that you had participated in the 'Voluntary Social Year' 
program? Why or why not? Knowing what you know now, would you still make the same 
decision? Why?” Six of the nine students expressed that they did not regret of not doing it 
because they knew what they wanted to do during that time, for example, studying at the 
university, working or/and traveling. One respondent stated that the wages of the social year 
were too low. Two of the nine students wished that they had participated in a social year, and one 
of them noted that the social year experience might have made him or her more mature. One of 
the nine students was not sure if in hindsight, he or she would have wanted to participate in a 
social year. 

After interviewing students from Germany about their time during their voluntary social 
year, their answers showed that, although the students participated in a variety of placements, 
whether abroad or more local, many of the answers provided similarities. 

When asked how students learned about the opportunity to participate in a voluntary 
social year, their answers varied from school teachers to siblings, to their parents, and even their 
peers. In some cases, their older siblings had already participated in the voluntary social year, 
which inspired the younger sibling to participate as well. Even though a few students explained 
that they learned about this opportunity from their schools, the students noted that their 
instructors mentioned the voluntary social year, but rarely went into detail about what it fully 
entails. Each student’s answer varied slightly, but overall, when asked why they had chosen to 
complete a social year, the word “independence” was overwhelmingly used in order to describe 
what they wanted to gain out of this experience. Similarly, many students spoke about their 
desire to experience something new, such as a new environment or a new culture, since most had 
lived in the same town for many years. 

In regards to how students chose where they were placed, not just their town, but also 
their placement in terms of work, there were a multitude of answers. One student recalled that 
their church provided a voluntary social year that was of interest to them. Another chose their 
placement due to their interest in a particular field, such as the medical field, in order to gain 
more experience for a future career. Others chose their placement either by previously working 
with the organization prior to participating in the social year or using one of the two main 
websites that provide information and placement opportunities for students after high school. 
The placements, in terms of their daily work, ranged from a youth hostel, a daycare facility, an 
emergency response team, and a school in which students participated in a plethora of daily 
activities. As mentioned previously, most students that were interviewed stayed in Germany for 
their placements, but two of the students traveled abroad, one to Bolivia and one to Brazil. 

When prompted to share details of their typical day while completing their social year, 
students provided many explanations. First, students discussed their hours and how long they 
would work each day which varied from 8-10 hours per day, 4-5 days a week, with the exception 
of the student that worked with the emergency response team in which she would sometimes 
work up to 12 hours a day, which would shorten the days per week she worked. On average, the 
students agreed they worked about 40 hours per week, similar hours to a full-time job. Second, 
the students were then asked to describe their daily routines and what their workdays consisted 
of. Students who worked in the daycare facility or in the schools worked with young children, 
mostly kindergarten, but some a little older. These students would provide the children with 
homework help, snacks after school, and other enrichment activities, which would be similar to 
an after-school program. One student provided assistance to a medical emergency response team 
in which she would ride in the ambulance with the other medical technicians and assist the team 
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in emergency situations. One student worked in a youth hostel in which he would design and 
provide games and enrichment activities for the students who stayed there. 

Next, the students were asked to describe how or if they were paid for their services, and 
if so, how much and in what form. For the students who were provided lodging (which consisted 
of usually an apartment or a flat), the students were paid less than if they were not provided with 
housing. However, given what was leftover from their payment, there was very little room for 
luxuries, and ultimately, after paying for food and other travels, the students were left with very 
little profit. The government, the church, a specific social year organization, or a combination of 
both were funding the students’ jobs and travel to their destinations. 

In addition to learning about what the students experienced while taking part in their 
social year, what the students felt they ultimately gained from this experience, whether it be 
rewarding or challenging, was of particular interest. Overwhelmingly, each of the students noted 
an increased sense of maturity and independence in addition to heightened self-confidence and 
knowledge of other cultures and customs. Students also felt that communication with strangers 
was easier, and overall this experience helped them better decide what steps to take next in terms 
of their future careers and/or life plans. Although most of the aspects of this year were positive, 
students that traveled abroad found it challenging to learn some of the customs and languages 
when first arriving. When working with children in low-income areas, the students felt that it 
was sometimes difficult to see children in less fortunate situations. The student that worked at the 
youth hostel also expressed a sense of loss of privacy since the students could essentially call on 
them at any moment to ask for assistance. This forced the student to essentially remain on-call 
during most of the evening and night. Another challenge that one student faced was the trauma 
that comes with being on an emergency response team and having to prepare to be present for 
potentially harrowing scenes. 

Fourteen students participated in the interviews, but little was known about how many 
students out of their graduating class participated in the volunteer social year. When asked, most 
students guessed around 10-15% of students in their class decided to complete a voluntary social 
year as opposed to going to university or enrolling in some other form of a training program. A 
student also noted that females were found more frequently among those working with younger 
children. When prompted to discuss how the participating students felt that non-participating 
students perceived the social year, 13 out of the 14 students said the non-participating students 
felt positive about it, although some may view the international travel as something to be 
concerned about. Only one student mentioned that perhaps a small percentage of non- 
participating students may not view the social year as beneficial due to the delay that it would 
cause in terms of starting university or a potential career. 

Reflecting back on their time engaging in their social year, positivity resonated through 
their answers. The majority of the students do not regret their time, and if given the chance, they 
would choose to participate once again. However, when asked if students would have done 
anything differently, the common answers seemed to be that they wished they would have stayed 
in placement longer so that they could absorb more in addition to taking more risks, and learning 
the language better prior to traveling for those who went abroad. 

Lastly, the students were asked how technology fits into their lives when conducting their 
voluntary social year. They all stated that technology played a critical role prior to, during, and 
after their social year. Prior to the social year, they searched for information about the voluntary 
social year using technology. During the social year, technology was used to communicate with 
family and friends. They also used technology to document their time spent away from home 


172 


through photos, videos, and digital journaling. After the social year, they used technology to 
keep connected with people who were part of their social year. In regards to what technology 
used, they expressed that they used a variety of devices, including their cell phones, ebooks, and 
laptops while using apps such as WhatsApp, social media, blogs, and Skype. 

The parents of a student who conducted her voluntary social year service in Germany 
provided insight into the perception that they have of their child’s participation in her social year. 
They echoed the positive impacts of the social year on their child: being more mature and 
independent, and knowing what she wanted to do after she returned from her social year. The 
parents pointed out that the voluntary social year in Germany is connected to the history of 
Germany. Males in Germany prior to 2011 were drafted into the military, but the option existed 
to claim to be conscientious objectors. Instead of service in the military, those conscientious 
objectors then had to participate in community service, which was similar to the service in 
voluntary social year. This connection to the history might explain why the volunary social year 
in Germany is so well organized and structured. 

The parents of another student who went abroad also provided insight into the perception 
that the parents have of their children’s participation in a social year. The parents found that their 
children, once back home, showed an increased sense of maturity and independence, which the 
parents found to be a very positive change. The children also showed more responsibility and a 
better understanding of what may come next in the child’s life. Additionally, the parents felt that 
because their children came back with the knowledge of other cities and cultures, this also 
translated over to the parents as well and they felt that they were able to learn along with their 
child. One parent, however, made an observation that little is said about the impact that the 
students participating in the social year have on their host families and communities, and it 
seems crucial to look into how this may affect them, as well. 


Limitations 


This study shed a good light on a voluntary social year in Germany and technology use 
by students who participated in the social year. At the same time, the authors identified the 
following limitations: 

The participants of the study were convenience sample, being recruited by two German 
high school graduates, one in the field of medicine and the other in teacher education. Thus, the 
majority of the participants were from these two fields of study: medicine and teacher education. 
Should the participants have been from other or a variety of fields of study, the results might 
have been different from what was found in this study. 

The study focused on perceptions of students who participated in a voluntary social year 
and also their parents’ perceptions. The study did not look into the organization of a social year, 
for example, how a social year program is organized and structured, or where the funding comes 
from. Visiting sites of the participants, for example, a daycare center or a youth hostel, will 
definitely give the authors a better picture of the life of the participants during their social year. 

The study indicated a positive impact of a social year on German high school graduates. 
It did not look into the impact of the German high school graduates on their hosting institutions, 
as one of the parents interviewed had pointed out. For example, what was the impact of the two 
German students, who went to Bolivia and Brazil, on the children in the two countries? Would 
the service they provided continue after they left Bolivia and Brazil? 
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Conclusions and Recommendations 


Based on the information that was gathered through the interviews and information 
provided by the students, it is evident that the volunteer social year is a positive experience for 
students. They expressed that they are in favor of the social year and that they have more 
maturity, more independence, and more confidence. The students also expressed an increase in 
knowledge of international values and customs which is beneficial. In addition, this study shows 
that students also claim to be better able to communicate with strangers and other adults, 
allowing them to be more open to difficult situations. Lastly, this information has also shown 
that students who come back from the volunteer social year have a better understanding, or idea, 
of what their future may look like once they return home. 

The parents who were interviewed also responded favorably to the social year, and it can 
be concluded that they find this experience to be beneficial for their children. It seems that the 
parents’ responses echo the sentiments noted by their children that there is a definite increase in 
maturity and independence when the students arrive home. The parents are overall pleased with 
the changes that they have seen in their children. 

In terms of technology, students communicated easier when able to use devices and apps, 
such as WhatsApp and FaceTime. Not only has the technology been beneficial in terms of 
communication, but it has also allowed for students to learn independently through the use of the 
Internet and other online resources. Technology gives students the opportunity to stay connected, 
share photographs, and document their journey which added to the novelty of their experience. 

Looking forward into future research, it would be wise to look into Germany’s history as 
to what the purpose of the volunteer social year originally was and how it became as 
commonplace as it is today. Looking into the history and the background, this process will 
provide a better understanding of the volunteer social year. Additionally, it would be interesting 
to look into how the host families of these students perceive this process and even more so, 
looking into the impact that the students leave on their host communities. Lastly, interviews and 
information from students and parents who did not complete the volunteer social year would also 
provide more insight into this experience and provide more information on how this process is 
perceived by others. 
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An Empirical Study on Using E-schoolbag to Promote Deep Learning in 
the Primary Mathematics Course 
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Abstract: It is believed that Information and Communication Technology (ICT) can 
promote deep learning, and E-schoolbag is widely used in primary and secondary schools in 
China to enhance deep learning in different subjects. There are nearly seventy students in 
two classes in the fifth grade of a primary school in China, and we designed quasi- 
experiment research to examine the use of E-schoolbag for promoting deep learning in 
primary school mathematics. The intervention condition was mainly the learning situation 
design supported by technology. Data were collected through experiments, interviews, and 
classroom observation. Through one month of teaching, the teaching of the experimental 
class and the control class were compared from the three dimensions of deep learning to 
draw relevant conclusions. It was found that E-schoolbag can promote deep learning in 
primary school mathematics, thereby optimizing the teaching situation of primary 
mathematics classroom and enhancing the effectiveness of teaching. 

Keywords: Information technology; primary school mathematics; deep learning; E- 
schoolbag 


Introduction 


Deep learning is the way of learning for decision making and problem-solving. 
Through the first cognitive experience, learners reorganize new things, integrate them into 
the unique cognitive structure (Entwistle, 2005). Learners also find many ways to connect 
or transfer existing knowledge into new situations. Constructivism provides a theoretical 
basis for deep learning that “learning depends on the individual’s construction of the 
meaning, the process of constructing meaning is embedded in a specific social context” 
(Gergen & Davis, 1985). The significance of constructivism in information technology 
education is that the task-driven, visual demonstration, operational practice, cooperative 
learning method, and other teaching methods are used in the curriculum to make deep 
learning more likely to happen. According to the age of students and the laws of cognitive 
development. The use of information technology has the potential to enable students to 
establish mathematical learning methods in specific situations to promote deep learning of 
primary school mathematics. 

E-schoolbag is not only a portable device, but also the learning platform and virtual 
learning environment for ICT in education, it was used to be called “E-textbook”, a 
comprehensive learning platform similar to a learning management system, that contains 
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digitized learning materials such as texts, videos, and audio clips and so on. In addition to 
the digitized learning materials, various tools were embedded in the E-schoolbag for 
facilitating teaching and learning activities such as assignment management, discussion 
space, collaborative tools for the group project, survey tools, tests, etc (Hui et al., 2017). 
The devices were connected to a wireless network, which makes it possible for creating a 
social environment where learners can grow together academically (Vygotsky, 1978). 

The author has participated in the project titled “Using Information Technology to 
Promote Effective Teaching Improvement” jointly operated by Ningxia Normal University, 
Guyuan Experimental Primary School, and TF Enterprise Research Institute, planning 
course with the primary school teachers, preparing lessons, and conducting classroom 
observations on the teaching of elementary school mathematics. It has been found that in 
the current mathematics classroom in elementary school, students only learned the 
mathematical symbol knowledge that the class teachers hope them to master. However, that 
is not the essence of learning. The information technology teaching equipment used was 
mainly E-schoolbags in the study. 

The use of E-schoolbag for promoting the deep learning of mathematics in primary 
schools has become a new research hotspot in recent years. However, the researches on 
information technology for promoting the deep learning of primary school mathematics at 
home and abroad basically focus on concept analysis and connotation understanding. In 
theoretical research such as strategy construction, empirical analysis is not in-depth (Huang, 
2015). In this paper, we took "deep learning" as the focus and used E-schoolbag to design, 
practice, analyze and evaluate the mathematics classroom in primary school. Through the 
analysis of the teaching classroom observation and test results, it is verified that E- 
schoolbag is useful to promote the deep learning of mathematics in primary schools, thus 
providing a more practical reference for K-12 teachers. 


Deep Learning 


“Deep Learning” as an academic concept originated in the United States in the mid- 
20th century. The research on deep learning mainly includes two fields: machine learning 
and cognitive science. In this study, deep learning is a concept developed from the field of 
cognitive science. It is also sometimes regarded by scholars as deeper learning. The 
National Research Council’s Committee on Defining Deeper Learning (2012) concludes 
that deeper learning is the process through which an individual becomes capable of taking 
what was learned in one situation and applying it to new situations (i.e. transfer) by 
developing cognitive, interpersonal, and intrapersonal competencies (NRCC,2012). The 
Hewlett Foundation (2012) defines deeper learning as the skills and knowledge students 
will need to succeed in a world that is changing at an unprecedented pace. Deeper learning 
prepares students to master core academic content, think critically, and solve complex 
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problems, work collaboratively, communicate effectively, and learn how to learn (for 
example, self-directed learning). Some researchers also believe that deep learning is a new 
learning method for decision making and problem-solving. The type of learning that results 
from students’ self-directed application of critical and creative thinking, problem-solving, 
communication, and collaboration to deepen their understanding of key concepts in the 
curriculum. Deeper learning is also the outcome of those processes (Bellanca, 2014). 

Combining previous scholars’ research on deep learning, the author figures that the two 
aspects including cognitive and non-cognitive factors these two aspects may influence deep 
learning, as shown in Figure 1. The cognitive factor includes relating new ideas and 
concepts to previous knowledge and experience, integrating their knowledge into 
interrelated conceptual systems, and looking for patterns and underlying principles, 
evaluating new ideas, and relate them to conclusions. Also, deep learning means that 
learners understand the process of dialogue through which knowledge is created and that 
they examine the logic of an argument critically. Non-cognitive factors mainly include 
students’ learning interest and attitude, as well as learning situations that can effectively set 
up for deep learning. The stimulation of learning interest is the key to successful teaching, 
which makes it easier for students to concentrate on the classroom and may benefit the 
exploration of new knowledge. Finally, it is possible to have deep learning. 

Participants in this study were children aged about 11 years old. According to Piaget's 
staged division of students' cognitive development, they are in the specific stage of 
operation. At this period, the children are freed from the apparent thinking, and their 
cognitive structure already has an abstract concept, so it can carry out logical reasoning. But 
the operation is still inseparable from the support of specific things (Shi, 2013). Although 
compared with the first two stages, children have made some progress, their cognitive 
ability is still weak, and it is difficult to establish links to connect the knowledge. Because 
of the age characteristics of students, while using information technology to promote deep 
learning in primary school mathematics, educators should pay more attention to establish 
the connection between various teaching contents. Teachers need to combine students' 
cognitive development rules and use information technology to create a kind of virtual 
reality life scenes for enabling them to establish mathematics learning methods in vivid and 
specific situations, fully mobilize students' multi-sensory participation, stimulate students' 
enthusiasm for learning, and change students' learning attitudes. Thus, promoting deep 
learning of mathematics in primary school. 

The construction of the learning context is an important indicator to measure the 
effectiveness of classroom teaching. Teachers use the skills of creating scenarios to enable 
students to learn effectively in the learning context. The effect of learning and the ability to 
transfer knowledge can better reflect the meaning of deep learning, which requires students 
to learn and apply abstract and complicated knowledge based on understanding, not simply 
through repeated memory. Studies have shown that the application of ICT can provide 
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certain situational support for the occurrence of deep learning (Peng, 2020). In this study, 
situational construction is mainly based on the E-schoolbag environment supported by 
information technology. 


@ relate new ideas and concepts 
@ integrate’ their knowledge into 
interrelated conceptual systems. 
@ look for patterns and underlying - Learning interest 
principles - attitude to learning 
@® evaluate new ideas and relate them 
to conclusions 


Figure1 Factors influencing the occurrence of deep learning 


Research Method 


This study adopted a quasi-experimental research design. The content of this study is 
aimed to confirm that information technology promotes deep learning in elementary 
mathematics. In this study, information technology mainly refers to E-schoolbag. The 
process is shown in figure2. 


Read related Search for literatures and then do deep reading, analyze, arrange the literatures, knowing the 
Propose To propose hypothesis with the consideration of primary school students learning of math on the 


Design According to the three dimensions of deep learning in the study, a comparative experiment was 
Collect Observe student’s classroom performance and collect test grades as well as the data collected from 


Analyze the To correspond the data with three dimensions of deep learning; and then the learning conditions of 
Get the The experimental results are obtained by analyzing the data and verifying the theoretical hypotheses 


Figure 2 Research process 
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Research Design 
Data source 

Two classes in the fifth grade in GY Experimental Primary School were enrolled in the 
study. In the mid-term exam, the mathematics scores of the two classes were comparable, 
providing pre-test data for further comparative studies. GY Experimental Primary School is 
currently one of the province's demonstration schools for information teaching. The school 
fully recognizes the important role of information technology in education and teaching, 
and actively promote ICT application in education, increases training on information 
technology applications for teachers, and actively uses multimedia. The collection and 
analysis of the data used in this study were carried out with the consent of the school. 
The framework of research design 

This study is designed from the three dimensions of cognitive ability, learning interest, 
and learning situation in deep learning. Firstly, data from observing the classrooms of the 
control class and the experimental class was collected, and then we have analyzed whether 
the situation constructed by information technology has a certain promotion effect for 
students; Secondly, some students from the experimental class and the control class were 
randomly selected to participate in the interview. The goal of selection was to understand 
the students' interest in learning mathematics and to compare the influence of different 
pedagogics on students' interest in learning. Thirdly, tests are designed for the course 
content and the test scores have been analyzed with SPSS software. The scores of the 
experimental and control classes were compared to see if there was a significant difference 
between the experimental and control classes, and further to infer whether the students have 
achieved meaningful learning or deep learning in the classroom. 

In this study, we set two classes. In the experimental class, E-schoolbag was adopted to 
support learning, the control class did not adopt the learning situation supported by E- 
schoolbag, E-schoolbags were mainly used as learning terminals for students. Besides, the 
teaching environment of both classes is the same, each student has the textbook, paper, and 
pen, and the classroom is equipped with a blackboard and an electronic whiteboard. 

The data of the cognitive changes of relevant knowledge points of students was 
collected through tests and then we obtained the non-cognitive data affecting the occurrence 
of deep learning through interviews and observation. 

Informational Instructional Design 

In the traditional classroom, students were taught in a passive position. Their 
knowledge is mainly derived from the instruction of teachers. In the 21st century, the 
unpredictability of mathematics classrooms in primary school has also increased 
accordingly. Teachers need to reconsider more factors and carry out a more specific 
instructional design. Therefore, to optimize the effect of information technology for 
promoting deep learning of mathematics in primary school, it is necessary to properly 
handle various contents such as textbook analysis, academic analysis, creation of learning 
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environment, teaching strategy, and teaching process. And then we have proposed a specific 
teaching design plan here as following: 
Analysis of teaching materials 

This study selected the content of the “Division” in the fifth grade of the Compulsory 
Education Curriculum Standard Experimental Mathematics Textbook. These students who 
take part in our research are 10-11 years old. The textbook contains two sections including 
the largest common divisor and the divisor. The divisor is the direct application of the basic 
properties of fractions, and it is a common method to simplify fractions. Learning about 
fractions can not only improve the understanding of the basic nature of fractions but also lay 
a foundation for learning the four operations of fractions. 
Analysis of the learning situation 

Considering that our participants are fifth-grade students, teachers should pay special 
attention to the specific reflection of the students while teaching, and adjust the teaching 
plan at any time during the whole class. The case in this study chose a total of seventy 
students from class 3 and class 4, most of the students were motivated to learn mathematics, 
who could acquire knowledge from the existing knowledge and experience, the abstract 
thinking ability has also developed during learning, the basic knowledge is relatively solid, 
there is the certain ability to learn mathematics. 


Results and discussion 


Classroom observation analysis and discussion 

Through observing the control class and experimental class, it was found that the 
traditional classroom and the information classroom showed significant differences in the 
expression forms of the teaching content, the teacher and student activities, the media 
application, and the teaching procedures as well as methods. In the series of traditional 
classroom activities featured with a lead-in by reviewing, teaching new lessen, 
consolidating and summarizing new courses-summary courses, the teaching content is 
simple, mainly based on imparting knowledge, occasionally joining discussions and 
questions, and rarely using information technology. Students are only passively accepting 
knowledge. However, in the series of informational classroom activities from Happy 
Review-Scenario, context lead-in, explore new knowledge, thinking expansion- 
consolidation to the summary, the teaching content is diversified. Teacher-student 
interaction and human-computer interaction are more obvious. The teacher plays the role of 
a leader and helper of student learning. The students are in the teaching situation 
constructed by information technology. Thus, their interest in learning is high, the 
motivation for learning is obvious, and the knowledge is acquired through group discussion 
and inquiry activities, which contributes to the knowledge content. Deep understanding will 
result in deep learning. 
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Interview data analysis and conclusion 

Interviewee: twenty students were randomly selected from class 3 and Class 4, grade 5 
of G Experimental Primary School. In this interview, 90% of the students in the 
experimental class reported that they like mathematics classes, because the application of 
various information methods enables students to actively participate in the classroom, and 
for some difficult knowledge point. Information technology can transform abstract 
mathematical symbols into visual and intuitive things, especially many animations that can 
show the mathematical operation process in the E-schoolbag, which makes students 
understand easily. Besides, the use of gadgets such as photographs and mutual evaluations 
increases the curiosity of students in the mathematics classroom, and the students' attention 
is more likely to focus on the classroom, which is conducive to student learning. 85% of the 
students in the controlled class say they don’t like mathematics, because mathematics 
knowledge is difficult, difficult to understand, and they are eager to communicate with their 
peers at any time to solve the problem and experience. In the context of informational 
teaching, the classroom tends to be transparent. Each participates, everyone is equal, and it 
can stimulate students' interest in mathematics. 
Test data analysis and conclusion 

The data was collected through two different treatments for two homogeneous samples 
to judge whether the results are different. Therefore, the data processing method of the 
paired-sample T-test in SPSS software is selected. The data is judged whether there is a 
significant difference in the mean of the paired population from which the sample is 
derived. Firstly, establish the null hypothesis HO:u1 =12, that is, it is assumed that there is 
no significant difference between the test scores of the experimental class and the controlled 
class. It means that the application of information technology means in the classroom does 
not have any influence on the student's learning. Secondly, the significance level of the 
hypothesis is determined to bea=0.05. The following is an analysis of test scores: 
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Table 1 Paired sample statistics 


Standard |The standard 
deviation |error of the mean 


Mean N 


Scores of — multiple choice in 
: 34.7714 |35 10.47149 | 1.77000 
experiment class 


S fauiltiple- choices 
Game saaicg tana ae 35.2857 |35 |12.13786 _|2.05167 
controlled class 


Scores of Ture or False questions in 
; 23.4286 (35 |6.39064 1.08022 
the experimental class 


N 


S f Ture or Fal Gonse 
See eee ee aoe eee 190.7143. 135° 1620043. [1.04806 
a controlled class 


Scores of subjective exercises in 
; 22.4286 (35  |4.59722 .77107 
experimental class 


S feubiscd an oh 
cores of subjective exercises in a 15.7143. [35 708591 (109g 
controlled class 


Total score in experimental class |80.6286 [35 16.07535 = |2.71723 
4 Total score in controlled class 71.7143 /35 15.66106  |2.64720 


As shown in Table 1. The average score of the experimental class multiple-choice 
questions is lower than the average score of the control class. The average scores of the 
Ture or False questions and the subjective questions in the experimental class are higher 
than that of the controlled class; accordingly, the average score of the experimental class is 
higher than the average score of the control class. It can be seen the two pairs of samples 
has a certain amount of change, but to determine whether the change is significant, or 
whether it is an essential difference, it is necessary to calculate the corresponding statics in 
two sets of samples.t statistic. 
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Table 2 Paired sample test 


Pairwise difference 


95% confidence 


interval for the 


class 


difference Sig. 
The (two 
standar sides) 
Standard Lower Upper 
Mean d error 
deviation limit limit 
of the 
mean 
Experiment class 
multiple-choice score 4.8788 
-.51429)15.70008 |2.65380 |-5.90745 847 
- control class 
multiple-choice score 
Experimental 
lass jud t - |2.7142 5.5805 
yee es 8.34397 |1.41039 |-.15197 063 
control class judgment 
score 
The subject class 
score of the 
experimental class- |6.7142 9.9928 
Res, ; 9.54424 {1.61327 |3.43573 .000 
the subjective subject 
score of the control 
class 
Experimental 
lass total - total }8.9142 16.538 : 
eae aii Ae ail 2219482 |3.75161 |1.29010 
score of the control 47 023 


Table 2 shows the key paired sample t-test results. The scores of multiple-choice tests 
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in the experimental class and control class were p = 0.847 > 0.05, which means that there is 
no significant difference between the two. Still, the experimental class judgment scores and 
the control class judgment questions. The score of p=0.063>0.05, there is no significant 
difference between the two; the score of subjective exercise of the experimental class and 
that of the controlled class are p=0.000<0.05, which indicates that in the case of the 
significance level of 0.05, the two-sided tests worth accompanying probability value 
p<0.05, so the null hypothesis HO should be eliminated and the alternative hypothesis 


H1should be accepted. Therefore, there is an obvious difference between the scores of 
subjective exercises of the experimental class and that of the controlled class; the total score 
of the experimental class and the total score of the controlled class are p<0.05, which 
indicate that in the case of a significance level of 0.05, the two-sided T-test worth 
accompanying probability value p<0.05, so reject the null hypothesis HO and accept the 
alternative hypothesis H1. Still, there is also a significant difference between the total score 
of the experimental class and the total score of the controlled class. 

According to the above data analysis results, in the classroom teaching with 
information technology, the accuracy rate of subjective question is relatively high, so that 
the overall performance of the class is better than the controlled class, which shows that 
students start their meaningful learning and generate deep learning. 


Limitations 


In the early stage of the study, although a large amount of literature was reviewed, a 
more comprehensive research design was carried out; the teaching classroom was analyzed 
and tested during the implementation process; the post-experimental data analysis was 
consulted by the teacher and teachers read many related books, there are still some limits 
which need to be further explored in the future. Firstly, the experimental samples in the 
empirical study relatively small, and the experiment was conducted on a tight schedule. 
Second, due to the limitation of manpower, material resources, and financial resources, the 
research only selected two classes. The effectiveness of information technology to promote 
deep learning in primary school mathematics needs to be tested, verified, and adjusted 
within a larger time and space. 


Conclusion 


Information technology is a developing concept. In the process of exploring 
information technology which can promote deep learning in primary school mathematics, 
by the observation record monthly and teaching effects evaluation on the students in class 3 
and 4 in fifth grade, it was found in the primary mathematics classroom that rational 
application of information technology can promote students' deep learning. 

The author believes that teachers should provide students with a good learning 
environment, create a reasonable teaching situation, and infiltrate lead-in, new teaching 
consolidation, and other parts of the teaching, to help students to sort out the relationship 
between various contents, thus achieving the task of completing teaching aims. However, 
the proper use of information technology in this process will produce a good impetus. 

Future research should examine the certain reference significance for the teaching of 
teachers teaching mathematics in primary school. The author calls on society, schools, 
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teachers, and parents to attach importance to the role of information technology, using 
information technology to promote primary school mathematics learning, and jointly 
construct information learning environments for students. And the final purpose is to 
cultivate students’ interest in learning mathematics, improve the effectiveness of students' 
mathematics learning, and better realize the aim of deep learning. 
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Introduction 


Problem solving consists of actions performed for a goal (Anderson, 1985). This ability 
has been at the core of mathematics education since it reflects the ability to cope with problems 
in and outside school (NCTM, 2000; Simsek, Uygun, & Giiner, 2020). The actions taken to 
solve a problem are related to cognitive processes (Metallidou, 2009). Researchers proposed 
various explanations for these processes and stages of problem solving. Lin and Lin (2014b) 
listed these processes that a solver must comprehend the initial text and the diagram, search for 
overlapping information with existing one, make a deduction, and do calculations to reach a 
solution. Correspondingly, Gal and Linchevski (2010) indicated that geometry tasks include 
consecutive phases from neural processes to higher cognitive ones such as reasoning and 
deduction. Epelboim and Suppes (2001) stated these stages, respectively, as reading the text, 
creating a diagram, looking for familiar patterns, recalling relevant information, and reaching 
a solution. Students might have difficulty while activating required cognitive processes in 
problem solving. Representations such as diagrams, graphs, tables, and charts are frequently 
used to help students to understand mathematical ideas within the problem solving context 
(Bolden, Barmby, Raine, & Gardner, 2015). It has been shown that diagrams presented by 
geometry problems aid students' understanding of mathematical concepts and problem 
statements (Brenner et al., 1997) and leading them to a solution (Greeno & Hall, 1997). They 
play a role in representing and explaining the mathematical ideas required to solve problems. 
Students should be able to gather properties and relevant information from the diagram and use 
them while solving geometry problems (Laborde, 2005). However, it is not certain that they 
can obtain the necessary knowledge (Pape & Tchoshanov, 2001), and they might have 
difficulty in making correct inferences during this process. 

Students use a variety of strategies during problem solving processes. This variation 
might derive from task characteristics (Gluck & Fitting, 2003). These strategies have different 
descriptions and classifications. Mayer defined a problem solving strategy as a guideline which 
not always lead to a solution (as cited in Gick, 1986, p.100). Dividing problems into parts is a 
common technique used in problem solving regardless of the domains. Another general 
strategy used by Simon and Newell (1971) is called as means-ends analysis. It includes filling 
the gap between the initial state and the aim of the problem. Assessing and evaluating students' 
problem solving processes provides information about their views about the context, 
difficulties, understanding, thinking methods, and strategies (Hegarty & Kozhevnikov, 1999; 
Simsek, Uygun, & Giiner, 2020). However, it is critical to select assessment tools to reveal this 
information. Prior studies frequently used self-reports and interview methods to assess problem 
solving processes and analyze students! strategies used during these processes (Huang, 2017) 


188 


and considered solution time and accuracy as problem solving measures which generate weak 
information about the internal cognitive process (Knoblich, Ohlsson, & Raney, 2001). 
However, it requires more influential techniques due to its complex nature. Eye tracking 
technology has the potential to observe problem solving processes in depth by capturing 
information people are attending during working on a task (Andra et al., 2015). The 
explanations based on eye movement measures might differ according to the context of the 
study. As an example, Duchowski (2007) specified that as the complication of problem 
increase, fixation number, and duration increase in that vein. However, problem difficulty 
cannot be the reason for the higher duration that individuals spend while looking at a point and 
higher times they look at that point in all conditions. In the study conducted by Lin and Lin 
(2014a), solvers good at geometry spent more time in an output area since they desired for 
more plausible answers. Therefore, it is essential to investigate the changes of eye movements 
measures and target specific explanations for these changes. Moreover, there have been various 
research studies about eye tracking and problem solving in different contexts; however, 
problem solving process itself needs to be investigated as Schindler and Lilienthal (2019) 
suggested. 


Methodology 


Eye tracking, which facilitates objective and assessment in an educational context was 
used in the study. The data of eye tracking was supported by Spatial Ability Test (SAT) results 
and transcripts of think aloud and interviews. The purpose of the study was to provide an 
overall analysis of the participants’ problem solving processes. Mainly, this study was 
conducted to examine the following research questions: 


e What are the problem solving strategies that high school students are using during 
problem solving in geometry? 

e How do eye movement measures change during problem solving processes at different 
problems and different parts of the problems? 


Eight high school students participated in the study with an approval received from 
Middle East Technical University Human Subjects Ethics Committee. Both students and their 
parents signed a consent form to specify voluntary participation and the approval that their 
children participate in the study. All participants first answered Spatial Ability Test (SAT) 
which measures participants’ spatial abilities. Afterward, they tried to solve six geometry 
problems on the computer. They were asked to think aloud, and their eye movements were 
recorded by Tobii X2-60 Eye Tracker during problem solving. In geometry problems posed by 
a diagram, two Area of Interests (AOIs), which are diagram and text, were defined. Diagram 
AOI implies problem figures, while Text AOI refers to written information related to the 
problem. The changes in specified eye movements measures, which are fixation count, total 
fixation duration, and visit count, were analyzed according to these two AOIs. Lastly, an 
interview was done with each student to examine problem solving strategies. Eye tracking data 
was used to decide whether eye tracking measures differ according to variation in problems 
and parts of a problem. The eye tracking data was compared with Spatial Ability Test (SAT) 
to make sense of participants’ eye movements. This data was also supported by think aloud 
protocol and interview method to determine participants’ problem solving strategies of 
geometry. 
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Findings 
Spatial Ability Test Scores 


Spatial Ability Test designed by Ekstrom et al. (1976) and translated into Turkish by 
Delialioglu (1996) is used in this study to assess participants’ spatial abilities. It includes four 
different tests, which are Card Rotation Test (CRT), Cube Comparison Test (CCT), Paper 
Folding Test (PFT), and Surface Development Test (SDT). First two tests are used to assess 
spatial orientation ability, while last two tests are used to assess spatial visualization. The 
results stated that Participant2 has the highest score in all results, while Participant5S has the 
lowest score. Participants’ Spatial Ability Test (SAT) scores indicated that the highest total 
score out of 282 is 267, which belongs to Participant2. On the other hand, the lowest total score 
is 105. Specifically, the scores range from 11-19 out of 20 in PFT, 26-58 out of 60 in SDT, 24- 
33 out of 42 in CCT, and lastly 38-158 out of 160 in CRT. The highest scores for PFT, SDT, 
and CRT belong to Participant2. Participant7 got the highest score for CCT. Participantl, 
Participant3, Participant4, Participant7, and Participant8 gave correct answers to more than 
half of the questions in all parts of the test. Participant5 did not answer all of the questions; in 
fact, she did not make any selection among drawings of a card within 80 items in the second 
part of the CRT. There are more unanswered questions than half of the items, which 145 empty 
questions out of 282. Participant6 has the lowest score for SDT. 


Problem Solving Strategies 


Table 1. Strategies used by participants 


Holistic Strategies Analytic Strategies 
Mereologic Visual Preproportional Qualitative Quantitative 
Way Strategy Reasoning Proportional — Proportional 
Reasoning Reasoning 
Participants 
Pl 4 2 2 
P2 3 1 3 
P3 1 1 1 
P4 FS) 2 1 
P5 Z 
P6 1 2 
P7 1 2 1 
P8 1 1 2 
Total 13 6 11 2 fi 


Note: The total number of questions asked in this study was 6. To solve one question, more 
than one strategy could be used. 


In response to the first research question, problem solving strategies were analyzed 
under two strategies, which are holistic and analytic strategies. Holistic strategies have two 
subcategories, which are mereologic way and visual in this study. On the other hand, analytic 
strategies have three subcategories, which are preproportional reasoning, qualitative 
proportional reasoning, and quantitative proportional reasoning. As seen in the Table 1, the 
most preferred strategy was the mereologic way. Preproportional reasoning was used at least 
one by all participants except P1 and it was the second most preferred strategy. Quantitative 
proportional reasoning was used seven times while qualitative proportional reasoning was used 
two times by only P8. Lastly, visual strategy was used six times. 
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Eye Movement Measures Changes 


To answer the second research question, eye movement measures, which are Fixation 
Count (FC), Total Fixation Duration (TFD), and Visit Count (VC), were analyzed. PFT and 
CCT, parts of SAT, were determined as covariates. The multivariate analysis results indicated 
a statistically significant difference of problems on eye movement measures, Wilk’s A = .484, 
F (15,221) = 4.432, p= .000. Also, a statistically significant difference of problem parts on eye 
movement measures was acquired, Wilk’s 4 = .424, F (3,80) = 36.195°, p= .000. The results 
are shown in Table 2. 


Table 2. The Effects of Problem Parts and Problems 


Effect Value F Hypot Error df Sig. Parti Noncent 
hesis al ; 
df Eta Paramet Observe 
Squa er d Power 
red 
Problem _ Pillai's Trace 576 36,195" 3,000 80,000 ,000 ,576 108,585 1,000 
Parts Wilks' Lambda 424 36,195> 3,000 80,000 ,000 ,576 108,585 1,000 
Hotelling's Trace 1,357 36,195 3,000 80,000 ,000 ,576 108,585 1,000 
Roy's Largest 1,357 36,195 3,000 80,000 ,000 ,576 108,585 1,000 
Root 
Problems Pillai's Trace 73 ~—- 3,871 15,000 246,000 ,000 ,191 58,072 1,000 
Wilks' Lambda 4484 = 4,432 15,000 221,246 ,000 ,215 60,509 1,000 
Hotelling's Trace ,951 4,985 15,000 236,000 ,000 ,241 74,779 1,000 
Roy's Largest 819 13,437° 5,000 82,000 ,000 ,450 67,186 1,000 
Root 


The results of between subject analysis to conduct the effects on each measure stated 
that there is a significant difference of problems on visit count, F (11,84) = 7.813, p= .000. 
However, there is no significant effect on fixation count, F (11,84) = 1.604, p= .168 and total 
fixation duration, F (11,84) = 1.508, p= .196. Between subject analysis also indicates the effects 
of problem parts on each eye movement measures. There is a significant difference of problem 
parts on fixation count and total fixation duration, F (11,84) = 63.620, p= .000, F (11,84) = 
60.247, p= .000 respectively. However, there is no significant difference on visit count, F 
(11,84) = .988, p= .323. The results of between subject analysis are shown in Table 3. 


Table 3. The Effects of Problem Parts and Problems on FC, TFD, and VC 


Source Depe TypellIl_ df Mean F Sig. Partial Noncent. Observe 
ndent Sum of Square Eta  Paramete d Power? 
Varia Squares Squared r 
ble 
Problem FC — 1180597,042 1 1180597,042 63,620 ,000 4437 = 63,620 1,000 
Parts TFD 85933,831 1 85933,831 60,247 ,000 4424 60,247 1,000 
VC 322,667 1 322,667 988 323 ,012 988 ,166 
Problem FC 148839,958 5 29767,992 1,604 ,168 ,089 8,021 oe 
S TFD 10757,239 5 2151,448 1,508 ,196 ,084 7,542 5504 
VC 12760,500 5 2552,100 7,813 ,000 3323 39,067 999 
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Regarding the problem parts, Table 4 showed that the means of diagram part were 
higher in all eye movement measures when compared to the text part. A statistically significant 
difference was obtained means between the diagram and the text parts on FC, p= .000 and TFD, 
p=.000. However, there was not a significant difference on VC, p= .323. 


Table 4. Comparisons of Problem Parts 


Dependent (1) (J) Mean Std. Sig. 95% Confidence 

Variable ProblemParts ProblemParts Difference Error > Interval for Difference? 

(I-J) Lower Upper 

Bound Bound 
FC diagrampart _ textpart 221,792" 27,807 ,000 166,475 277,108 
textpart diagrampart -221,792" 27,807 ,000 -277,108 -166,475 
TFD diagrampart _ textpart 59,838" 7,709 ,000 44,502 75,174 
textpart diagrampart -59,838° 7,709 ,000 -75,174 -44,502 
VC diagrampart _ textpart 3,667 3,689 323 -3,672 11,006 
textpart diagrampart -3,667 3,689 323 -11,006 3,672 


Based on estimated marginal means 
* The mean difference is significant at the ,05 level. 
b. Adjustment for multiple comparisons: Bonferroni. 


Regarding the problems, the results indicated that there was a significant difference in 
visit count between Question | and Question 2, Question 2 and Question 3, Question 2 and 
Question 4, Question 2 and Question 5, and Question 2 and Question 6. As mentioned above, 
there were no statistically significant effect of problems on fixation count and total fixation 
duration. Therefore, Table 5 shows the results of visit count. 


Table 5. Comparisons of Problems 


Dependent (I) (J) Mean Std. — Sig.® 95% Confidence 
Variable Problems Problems Difference — Error Interval for Difference’ 
(I-J) Lower Upper 
Bound Bound 
VC Question! Question2 -32,938° 6,390 000 -52,258 -13,617 
Question3 -8,750 6,390 1,000 -28,070 10,570 
Question4 -10,125 6,390 1,000 -29,445 9,195 
Question5 2,500 6,390 1,000 -16,820 21,820 
Question6 -5,813 6,390 1,000 -25,133 13,508 
Question2 Question! 32,938" 6,390 000 13,617 52,258 
Question3 24,188" 6,390 004 4,867 43,508 
Question4 22,813" 6,390 009 3,492 42,133 
Question5 35,438" 6,390 =,000 16,117 54,758 
Question6 7125. 6,390,001 7,805 46,445 
Question3 Question! 8,750 6,390 1,000 -10,570 28,070 
Question2 -24,188° 6,390 004 -43,508 -4,867 
Question4 -1,375 6,390 1,000 -20,695 17,945 
Question5 11,250 6,390 1,000 -8,070 30,570 
Question6 2,938 6,390 1,000 -16,383 22,258 
Question4 Question! 10,125 6,390 1,000 -9,195 29,445 
Question2 22813" 6,390 009 -42,133 -3,492 
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Question3 1,375 6,390 1,000 -17,945 20,695 


Question5S 12,625 6,390 = ,773 -6,695 31,945 
Question6 4,313 6,390 1,000 -15,008 23,633 
QuestionS Question1 -2,500 6,390 1,000 -21,820 16,820 
Question2 -35,438° 6,390 000 -54,758 -16,117 
Question3 -11,250 6,390 1,000 -30,570 8,070 
Question4 -12,625 6,390 = ,773 -31,945 6,695 
Question6 -8,313 6,390 1,000 -27,633 11,008 
Question6 Question 5,813 6,390 1,000 -13,508 29,133 
Question2 -27,125° 6,390 001 -46,445 -7,805 
Question3 -2,938 6,390 1,000 -22,258 16,383 
Question4 -4,313 6,390 1,000 -23,633 15,008 
Question5S 8,313 6,390 1,000 -11,008 27,633 


Based on estimated marginal means 
* The mean difference is significant at the ,05 level. 
b. Adjustment for multiple comparisons: Bonferroni. 


Moreover, heat maps which are powerful visualizations for visual attention were 
investigated. As an example, Figure | shows the participants’ focus for Q1 and Q2. Heat map 
of QI indicates that participants mostly looked at the diagram part compared to text part. Heat 
map of Q2 specifically indicates the focus on the base of the triangle while the focus is gathered 
on the line segment AC and also the point F located on this line segment in the diagram part. 


Figure 1. The participants’ focus on Q1 and Q2, respectively 


Limitations of the Study 


This study is limited to one city of Turkey and only high school students who 
volunteered to participate in the study were targeted. There might be an eye tracking effect 
since students were aware that they are being recorded. 


Conclusion and Discussion 


The findings of think aloud protocol and interviews indicate that students use different 
strategies during solving process. The mereologic way was the most preferred strategy, while 
qualitative proportional reasoning was the least. This variation in strategies derives from 
problem characteristics (Gluck & Fitting, 2003), difficulty, and complexity (Gitimu & 
Workman, 2007). While the use of strategies led them to a solution, they sometimes had 
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difficulty in solving processes. In some cases, students could not reach the correct answer since 
they missed crucial geometric information to apply their strategies as Lin and Lin (2014b) 
stated. Another reason that some students could not be successful while applying a strategy is 
attributing some label to geometric figures without proof or justification. For example, 
assuming a line is perpendicular or a triangle is an equilateral or isosceles without analyzing 
well and correctly resulted in incorrect solution. Moreover, participants repeated that they are 
not able to remember the question type and how to solve these types of question. It means that 
they focused on the way of solution instead of analysis of the question or use of geometric 
knowledge. 


The eye movement measures findings state that participants have higher fixation count, 
total fixation duration, and visit count in diagram part compared to text part, after controlling 
Paper Folding Test and Cube Comparison Test scores. The finding is in line with the results 
which proposed that participants had longer fixations to the diagram since they required to 
reread the diagram when they felt difficulty (Lin & Lin, 2014a). Just and Carpenter (1976) 
suggested that more fixation count is related to cognitive processes, in this regard extracting 
necessary information from the diagram to solve a problem can be resulted in increasing focus 
on diagram. Questions asked in the study required modifying diagram such as dividing the 
object into parts or exploring geometric relationships such as proportionality. Additionally, as 
mentioned above, some students attempted to recognize and remember the question type while 
looking at the diagram. Therefore, it is reasonable to obtain higher fixation-based measures. 
The present study is also in accordance with results that different Area of Interests and 
problems effect fixation count (Lin & Lin, 2014a). 
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Abstract 


This qualitative study tends to explore what accessibility design can be most important to 
facilitate learning in an online course for postsecondary students with low vision. The study is 
conducted in a U.S. public university offering online courses in the Midwest. This study was 
guided by two research questions: (1) What accessibility design did students with low vision 
who experienced online courses perceive to be helpful for their learning? (2) What accessible 
features would students with low vision want to exist in future online courses? The theoretical 
framework for this study was Universal Design for Learning (UDL). Three participants were 
interviewed to share their experience with online learning and to explore which accessibility 
aspects were perceived the most helpful for students with low vision. The finding revealed that 
alternative formats for materials—such as Word documents or Rich Text formats (RTF) and 
accessible PDF files—were the most helpful accessible text format in the online courses. In 
addition, the finding showed headings and color contrasting for the online content are the main 
aspects of design to increase accessibility and to facilitate reading for students with low vision. 
The last finding revealed that students with low vision need two additional accessibility design to 
be employed in online courses: audio response and instructor video. 


Keywords — Online learning, Accessibility, Low-Vision, Visual impaired 


1. Introduction 


In the last two decades, due to flexibility and accessibility, online courses are becoming 
increasingly popular among non-traditional students and learners who have (in)visible disabilities 
(Summers et al., 2014). Students with disabilities need support, particularly, students with low 
vision need special accessible features in online courses because they have some difficulties 
dealing with the technologies (Crow, 2008; Fichten et al., 2009; Summers et al.,2014). Universal 
Design of Learning (UDL) provides several accommodations for postsecondary students with low 
vision through using assistive technology and providing a variety of accessible features for this 
type of vulnerable population (Crow, 2008). Relevant research (Lorenzin & Wittich 2019; Okiki, 
2019) shows that low vision students will succeed academically when they take online courses 
with proper accessibility design. To explore which types of accessibility design aspects, based on 
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the UDL principles, are deemed to be appropriate for online settings, this qualitative study seeks 
to understand the perceptions of students with low vision. 


2. Literature Review 
2.1.Online Courses 

Online courses attempt to create a type of learning environment and serve as a process of 
connecting students, instructors, and learning resources when they are not physically present in the 
same location (Park & Choi, 2009). In 1997, the first online course platform was launched at 
famous universities, such as Yale, Cornel, and University of Pittsburgh. In the same year, a 
Learning Management System (LMS) known as Blackboard™ was founded and has become 
widespread to deliver online instruction and it is still utilized in many educational institutions and 
universities across the globe (Morton, 2016). Online courses use asynchronous and synchronous 
technologies. Synchronous technology requires students and instructors to work at the same time 
but not in the same place through using video conference (Palmer, 2012). In contrast, asynchronous 
technology does not require students and instructors to work at the same time (Palmer, 2012). 
They can work independently at a convenient scheduled time for each of them. 

In the last three decades, online courses have significantly increased in higher education 
(Betts et al., 2013). Recently, 30% of postsecondary students are enrolled in at least one online 
course in one of the U.S higher education institutions (Cole et al., 2014). Although online courses 
have increased, students with disabilities enrolling in institutions of higher education have also 
increased over the last twenty-five years (Lyman et al., 2016). Higher education has attempted to 
make online courses more effective and accessible for all students, however, some instructors 
and/or institutions may overlook the needs of students with disabilities (Kharade & Peese, 2012). 
Cook and Gladhart (2002) stated that 10% to 15% of postsecondary students identify themselves 
as disabled. According to the American Disabilities Act (ADA), a disability is a physical or mental 
impairment that substantially limits one or more major life activities. To be labeled as disabled, a 
person must have a history or record of such an impairment, or a person should be perceived by 
others as having such an impairment. These self-identified students with disabilities should have 
equal opportunities in their online courses as other students. 


2.2.Online Courses for Students with Low Vision 

Low vision is one of the common types of visual disabilities (Richardson, 2014). It is 
defined as the functional limitation of the eye or eyes or the vision system (The American 
Foundation for the Blind, 2015). The American Foundation for the Blind (AFB) defines low vision 
as a condition caused by eye disease in which visual acuity is 20/70 or poorer in the better-seeing 
eye and cannot be corrected or improved with regular eyeglasses (AFB, 2015). Students with low 
vision usually have several academic difficulties (Moola, 2015). One of these difficulties is using 
technology because sometimes they cannot adjust technology according to their needs. For purpose 
of this study, low vision identifies as “a person who has difficulty accomplishing visual tasks, even 
with prescribed corrective lenses, but who can enhance his or her ability to accomplish these tasks 
with the use of compensatory visual strategies, low vision and other devices, and environmental 
modifications” (Corn & Koenig, 1996, p.4). 

Consequently, the emergence of online courses has brought challenges for students with 
low vision (Argyropoulos et al., 2019; Summers et al., 2014). The literature on the experiences of 
students with low vision is scarce, and most seminal articles focus on students with disabilities 
without specifying the type of disability (Lorenzin & Wittich 2019; Okiki, 2019). However, some 
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relevant studies (e.g., Lee & Oh, 2017; Richardson, 2014) had observed that students with low 
vision are not often active in online courses due to the challenges they face in interacting with 
learning materials. On the contrary, several studies suggest online courses are beneficial for 
students with low vision since they provide remote learning experience (Barnard et al.,2012; 
Haegele et al., 2018; Kharade & Peese, 2012) and allow instructors to provide remote instructional 
assistance to the students anytime and anywhere even if they live far from the main campuses of 
the universities (Holmgren, 2018). 

Online courses benefit students with low vision because they can find a solution for the 
challenges attending physically on campus, which poses great difficulty for them (Kharade & 
Peesa, 2012; William et al., 2006). Kharade and Peesa (2012) addressed that the flexibility in the 
location, scheduling, and delivery of online courses reduced the challenges for attending on 
campus by providing flexibility in time and place of delivery. Feucht and Holmgren (2018) 
reported that students with low vision drop out because they cannot drive to the campus and do 
not live close to the campus. Walking around campus is also a challenge because sometimes it 
requires students with a very low vision to use aids such as a cane or a guide dog. This is because 
in some cases students with low vision cannot even see the small things, or in other cases, students 
with low vision cannot see things in bright or dark places. Therefore, they often have a difficult 
time self-navigating outside of well-known environments and prefer to be indoors (Long et al., 
1990), and some prefer to study and work in small physical spaces (Haegele et al., 2018). Asa 
result, low vision affects a person’s ability to learn or perform many job duties, which severely 
limits his/her main life opportunities for education and employment (Long et al., 1990). Therefore, 
online courses became such a great option for students with low vision to complete their 
educational degrees and be more motivated to succeed (Kharade & Peese, 2012). 

Besides flexibility, online courses allow students with low vision to adjust the instructional 
material through assistive technologies according to their needs (Crow, 2008; Fichten et al., 2009) 
during learning, reading, writing and acquiring academic and nonacademic skills (Hewett et al., 
2017; Rosner & Perlman, 2018). In addition, using assistive technologies in online courses help 
students facilitate learning and receive equal learning opportunities (Hewett et al., 2017). Because 
of this equality, students with low vision can be more active and motivated to participate in online 
activities such as discussion and group work. Assistive technologies help to improve the quality 
of learning for students with low vision (Crow, 2008). Online courses with assistive technologies 
encourage students with low vision to be active participants and share the ideas with classmates 
and instructors remotely in online course activities (Crow, 2008; Fichten et al., 2009; Hewett et 
al., 2017). 


2.3.Accessibility 

Accessibility is an important priority in online courses delivered by top universities such 
as Harvard University, UC Berkeley, and MIT (Alahmadi, 2017). Following their trend, many 
colleges and universities have started to make program and policy changes in their online courses 
(Zuriff, 1996). Accessibility addresses the design of technology rather than the needs of specific 
individuals (Alahmadi, 2017). Accessibility means using course materials and tools by all types of 
students, regardless of their physical and/or developmental impairments. When a course is 
accessible, most of the students even those with disabilities can reach the material equally. All can 
get access to the course delivery system, navigate the course content, submit assignments, and 
successfully use all course tools. The most common example of accessibility includes obtaining 
printed materials in alternate formats (Pittman, et al., 2014). Other examples involve the inclusion 
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of a statement of support for students with disabilities in the course syllabus. In addition, all video 
content (web, DVD, and VHS) should be captioned, and transcripts of audio-based material and 
video-based materials should be available if they cannot be captioned. 

There are several benefits of accessibility in online courses. The accessibility allows 
students to use flexible materials that can be adjusted according to their special needs and 
preferences (McKenna & Velasco, 2018; Pittman, et al., 2014). Audio, images, graphics, 
animations, video, or text which are often the tools to present information and the relationships 
between objects, actions, numbers, or events. However, visual representations are not equally 
accessible to all students, particularly visually impaired students (McKenna & Velasco, 2018). 


3. Theoretical Framework 

Most institutions of higher education in the U.S. incorporate the principles of Universal 
Design for Learning (UDL) into the educational and instructional materials. UDL is a framework 
for improving instruction because it helps provide equal opportunities for all learners to succeed. 
This strategy provides flexibility in how learners to access, engage with and demonstrate what they 
understand and increases the quality of learning materials for everyone (Rose & Mayer, 2008). 
UDL principles support students with low vision who have some challenges in online courses by 
providing resource and flexibility access to engage the students complete learning (Houston, 
2018). Most of the research has found that UDL is essential for integrating students with visual 
impairments into higher education (Al-Azawei et al., 2016; Houston, 2018; McKenna & Velasco, 
2018). According to CAST (2008) and Rose and Mayer (2008), there are three UDL principles: 
representation, action and expression and engagement. The first principle of UDL is 
“Representation, ” which involves providing learners with various ways of acquiring information 
and knowledge that have a connection to the accessibility formats, which require instructors to 
provide various resources to facilitate students’ access to the learning materials. The second 
principle is “Action and Expression,” which provides students with various routes to access the 
necessary materials using assistive technology. The third principle is “Engagement,” which 
enables an instructor to tap into students’ interests, challenges them appropriately, and motivates 
them to learn through facilitating resources. 

This study sought to explore the perceptions of the current experiences of students with 
low vision in online courses to identify what accessibility design aspects offer the greatest support 
based on UDL guidelines and would be beneficial. This research is intended to provide 
recommendations for future instructors and instructional designers to consider when creating 
online courses for students with low vision. 


4. Methodology 
4.1.Setting, Sample and Participants 

The study took place at a U.S. public university in the Midwest with a total enrollment of 
students 17,169 for Fall 2018. According to the university website (2018), there are 12,788 
undergraduates, 4,121 graduates and 260 college of law. As the mission states, the school 
celebrates its diverse population in all its forms, including gender, race, ethnicity, ability, 
spirituality, sexuality, age, and individual identities. This Midwest public university offers 
approximately 20 undergraduate and graduate degrees online and about 10 additional certificates 
fully online. 

For this study, purposeful sampling was used because the researcher purposefully selected 
the students with low vision who were enrolled in online course settings. Creswell (2009) stated 
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that in qualitative research, researchers identify participants and sites using purposeful sampling 
based on places and people that can best help a researcher understand the central phenomenon. 
The participants were from different major and educational level and ages range from 22 to 54 
years old. All the participants registered in the disability resources center (DRC) and had number 
of online courses taken between 3 to 6 courses. The participants had different low vision types; 
Retinitis Pigmentosa, Optic Nerve Coloboma, and Blurred Vision. 


4.2.Research Questions 
1) What accessibility design did students with low vision who experienced online courses 
perceive to be helpful for their learning? 
2) What accessible features would students with low vision want to exist in future online 
courses? 


4.3.Data Collection 
To obtain data, three students were interviewed, and the interviews were audio-recorded. 
Interviews were conducted either face-to-face or by telephone depending on the participants’ 
preference. The duration of each interview was 45 to 60 minutes. They were asked around 20 
questions, consisting demographic questions, questions related to their experience with online 
courses, and questions regarding accessibility and assistive technologies that helped them 
overcome their challenges. 


4.4.Data Analysis 

Using a professional transcription service (Rev.com) to transcribe the interviews. Then, the 
textual data of the interviews has been read multiple times to gain a deeper understanding on 
information contributed by participants (Creswell, 2012). Prior to proceeding to the data analysis, 
a codebook was created based on each research question: accessibility coded as AA and 
highlighted in pink; assistive technology coded as AT and highlighted in blue; and the wish list of 
the students with low vision coded as WLA and highlighted in red. The data had been analyzed 
line-by-line to code thoroughly. Then, applying an open-coding strategy to analyze the responses 
from the interviews by looking specifically for words that are related to pre-defined codes. For 
example, when the participants mentioned a screen reader, the researcher coded it as AT and 
highlighted it in blue. According to Patton (2002), the process of coding starts with segmenting 
and labeling similar codes to form descriptions and broad themes. Therefore, each of the pre- 
defined codes labeled as themes. The themes were used to respond to each research question. 


5. Findings 
5.1.What accessibility design did students with low vision who experienced 
online courses perceive to be helpful for their learning? 


The participants identified three accessibility design aspects: alternative formats for 
materials, headings, and color contrasting for online content as the most helpful accessibility 
regarding their learning experiences. Ruby responded that she has a lot of reading requirements 
to complete the online assignments as a graduate student. She clarified, ““Word document and RTF 
are the most beneficial types of alternative formats for textual online materials. Having formats 
like word documents or RTF are super helpful to access the text and use Read Aloud feature when 
I need it.” 

Sarah added, “I avoided reading. I avoided doing anything that wasn't just hands-on like 
the building because I'm actually in the construction trades.” Also, she mentioned that she has 
difficulty reading a book but she can read text font “Arial’ and size 16 or 18. She said, “sometimes 
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I get notes, teacher's notes and IJ actually have to change the font to just an Arial font because the 
New Times Roman is hard for me to read. Yeah, it has too many like little curves in it and the 
letters are too close.” Thus, she requests alternative formats for the online textual materials to 
audio: “Well, everything needs to be audio for me.” 

In contrast, Karen described that she prefers Adobe accessible PDF version, which includes 
features that allow students with low vision access text to be more readable such as taking notes, 
searchable text, and tracking of information. 


I would like to see that used more and more like accessible PDF documents instead of just 
like taking a picture. I think it's just giving me more access. For example, being able to 
look at like a PowerPoint in an accessible format. I can use it more easily to take notes 
and to keep track of information instead of having to like struggle through the slide. 
Sarah and Ruby use screen reader software, which they mentioned as the most helpful assistive 
technology for them. Ruby said: 


And so being able to have, like for example, the articles that we had to read were in two 
formats. They were in like a scanned in PDF and they were in like a word document. And so I 
was able to use my screen reader to read the word document and I had access to the course 
material without having to ask somebody to help me read it or help me scan it and to be able 
to change the scanned document. 


Moreover, Sarah and Ruby use the “Read Aloud” feature in Word. For the web pages and other 
documents, they use screen reader software “Narrator” in Windows. They mentioned that they do 
not install JWAS or Kurzweil 3000 on their computers. Ruby has used JAWS in the past; however, 
she is not using JAWS anymore because the screen reader is available in Windows and helps her 
read long articles. 

Additionally, the participants identified heading and color contrasting for online courses 
that helped them find and use online course materials. Ruby and Karen mentioned that designing 
online courses with headings guide them during navigating the online environment. Ruby said, 
“T'd say headings help split things into sections.” Karen said, “have a lot of headings to navigate 
that makes things easier.” In addition, Sarah mentioned that using contrast color for the text and 
background facilitates reading the PowerPoints. She described her current challenge reading some 
colors: 


One of the classes I have now, he has a lecture, it's short, it's a simple lecture. But then he 
has a PowerPoint and that does not have any audio to it and it's kind of a struggle for me 
to read through that. It's actually on the university [brand]; the background is red with the 
black and the gray and that's actually hard to read. I think that's hard to read. 


An additional accessible design that one of the interview participants reported was added 
description for videos. Sarah said, “I'd say more ideal description from videos” would help students 
with low vision understand what is happening in the video. 


5.2.What accessible features would students with low vision want to exist in future 
online courses? 


The results revealed some of the improvement accessibility suggestions that students with 
low vision wished would exist in online courses. Audio and instructor video were the two 
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alternative formats that did not exist in most of online course. The participants described two ways 
of using audio in online courses: audio with PowerPoint and audio response in the discussion 
boards. Sarah suggested that instructors in online courses should use audio with PowerPoint slides 
to facilitate learning; she said “when the teacher has a PowerPoint, it's great, but I have to read it. 
I want it to read to me and I want the word to stand out as are being read.” Ruby suggested adding 
the option of “audio” to participate in the discussion boards; she said: 


I think more audio would be really helpful. So I don't know if this is something that 
necessarily instructors would have jurisdiction over, but I guess just having it's different 
alternatives, communicating with discussion boards, you know maybe having like an audio 
option to leave audio responses. They're having a more simplified platform. So that would 
be one of the things on my wish list. 


Additionally, all the participants suggested that instructors in online courses should record 
videos to help the students be engaged in their learning. Karen would have more videos to 
understand some subjects; however, she did not specify the video types. She said, “Say we were 
assigned to read a chapter and then the teacher would have provided a video or something 
explaining certain things. I think that's always helpful.” On the other hand, Ruby and Sarah 
specified to receive instructor-recoded videos. Ruby said, “I wish the professors would do in- 
person videotaping of themselves.” Moreover, Sarah added that seeing the body language helps to 
engage the students in online courses; she said, 


When the instructor goes into the connect and has a PowerPoint and he's just talking, I 
think that's okay. But it would be really nice if you actually saw him because movement, 
your body gestures are engaging. 


Sarah also recommended the instructors should record video to explain the course content to 

improve the students’ learning performance; she said, 
Everything was online. It would have been so cool if the teacher had done what my 
classroom teacher did and said, “Okay, here's 20 minutes, here's the problem on the 
chalkboard. I'm video recording myself and this is what you do and, oh, you think about 
this and now, you go to the next step and you have to remember that. And then ...”’ That 
would have been great. I mean I know that some minor technical classes, safety will say, 
“Look, my last class was a safety class.” I mean, I can't imagine an instructor going, if I 
taught the class ... Ultimately I would like to teach but if I taught the class, I probably 
would read the book, they're like safety hazards. So let's say in the parking lot or in a 
building, I probably would record it. “This is a safety hazard.” 


6. Discussion 
6.1. Accessibility 


There were three major findings related to accessibly. The first finding revealed that 
alternative formats for materials—such as Word documents or Rich Text formats (RTF) and 
Adobe accessible PDF files—were the most helpful accessible format in the online courses. These 
alternative formats allowed students with low vision to make changes according to their own needs 
and use text-to-speech assistive technology such as screen reader software or the “Read Aloud” 
Word file feature. All participants highlighted that the alternative formats provided them equal 


203 


access to the online materials. In addition, this finding is consistent with literature (e.g. Pascual, 
2014; Spooner, 2014) mentioned students with low vision preferred to use alternative formats 
because it allows them to edit and make changes that best suits their needs. For example, Sarah 
explained that she could only read the “Arial” font; therefore, having the materials in Word allowed 
her to change the font to “Arial” because “Times New Roman” was hard for her to read. This 
finding is consistent with Houston’s (2018) study that recommends using Sans-Serif fonts in online 
course materials because Serif font types help make online content more legible to all students, 
including those with visual impairments. Common Sans-Serif fonts include Arial, Trebuchet, and 
Helvetica. On the other hand, some Serif font types—such as Times New Roman, Courier, New 
Century Schoolbook, and Palatino—have semi-structural details or small decorative curves on the 
ends of some of the strokes, making the letters and symbols challenging to read. Overall, this 
finding aligns with UDL “representation” principles, which entail the accessibility of instructional 
materials for all students, including students with disabilities, providing them equal access (CAST, 
2008). 


The second finding was that participants identified assistive technology within alternative 
formats as one of the most helpful accessibility features. The finding showed that text-to-speech 
reader software, such as screen reader, was the most helpful assistive technology for students with 
low vision to read online course materials. For instance, Sarah mentioned that she always needed 
to use assistive technology such as a screen reader to convert the text materials to audio. Also, 
other participants mentioned that using a screen reader reduced their challenges when reading 
online materials, as they did not have to seek assistance in reading the materials. In addition, the 
finding showed that Word processing was the most helpful assistive technology because of its 
“Read Aloud” feature. These findings are consistent with the literature that identified text-to- 
speech assistive technology as the most beneficial to suit the individual needs of students with low 
vision (e.g., Hersh & Johnson, 2010; Fichton et al., 2009; Nees & Berry, 2013). In addition, this 
finding is tied to the “action and expression” UDL principle, which asserts that individuals with 
disabilities should get opportunities for independence through the use of assistive technologies as 
they help them in overcoming barriers in the educational environment (CAST, 2008). 


However, the findings of this study were not consistent with some literature on magnifiers 
as helpful assistive technology. The participants expressed that they have visual condition abilities 
to read the original document. Karen mentioned that magnifying or enlarging text were not helpful 
for her in online courses. Sarah mentioned that she avoids reading and she prefers using speech to 
text assistive technology to receive information through audio mode. According to this finding, 
magnifiers as the main method of providing accommodations were less useful than the ability to 
choose which forms of assistive technology were most beneficial for the online courses. 
Therefore, the participants needed to hear the information or conversations in online courses, so 
they did not need to use a magnifier but use speech to text features such as “Read Aloud” or screen 
reader. This finding evidence that online course designers should provide more accessible types 
of online materials to allow individuals to choose which type of assistive technology will work 
best for them such as text-to-speech. 


The third finding showed two aspects of design to increase accessibility and to facilitate 
reading for students with low vision: headings and color contrasting for the online content. The 
participants indicated that headings is helpful to direct their attention toward key concepts and 
facilitate navigation; however, the study did not reveal adequate headings styles for students with 
low vision. This finding is consistent with literature (e.g., Kearns et al., 2013) that recommends 
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online course designers to design online materials with headings and use high-contrast colors, plain 
backgrounds, and scalable text for low vision or colorblind students as they allow them to skim 
the page quickly. Headings allow students with low vision to locate the information more easily 
and grasp the main ideas of the text (e.g. Fichten et.al., 2009; Houston, 2018). 


Online materials with low contrast can be difficult to read for students with low vision, 
making color contrast necessary to improve accessibility (e.g. Houston, 2018). Sarah mentioned 
that she had difficulty reading the online PowerPoint slides because of the black and gray text font 
contrasted in a red background. She expressed this background color was not suitable for her 
visual condition. However, this study did not expand on color contrasted of online materials. 
Houston (2018) suggests avoiding some color combinations that are not easy to read for students 
with low vision, such as blue links on black backgrounds, red text on green backgrounds, or other 
combinations where contrast is not enough. Although, Houston (2018) study did not find a list of 
color combinations that can assure accessibility for students with low vision, his study suggests 
that materials in online courses should be presented using a dark font color contrasted with a pale 
background. 


Finally, UDL principles and literature (e.g. Kharade & Peese, 2012; Pittman & Heiselt, 
2014) address additional beneficial accessibility aspects for students with low vision, but the 
findings of this study showed some of these aspects were not applicable in participants’ online 
courses. Aspects that were not present in online courses for most of the participants in this study 
were closed captions on video media; a transcript of the video or audio presentation; visual analogs 
to represent emphasis and prosody (e.g., emoticons, symbols, or images); and text descriptors for 
any relevant image, graph, or chart. However, Sarah mentioned that added description for videos 
was helpful to understand the video contents. This finding supports UDL representation principle 
which suggests that presenting information in several formats increase accessibility. 


6.2. Wish List for Students with Low Vision in Online Courses 


The findings revealed that students with low vision need two additional accessibility design 
aspects: audio response and instructor video. Students with low vision can benefit from submitting 
their responses to discussion board as audio files. This would reduce time spent on formatting the 
answer, such as using a screen reader to double-check response and allow them to focus on content. 
This finding is consistent with (e.g. Ching & Hsu,2015) that addressed audio discussion modality 
in online courses; however, the literature does not address the needs of students with low vision. 

In addition, the participants expressed the need for videos in which their instructor presents the 
content. This finding is consistent with the literature (e.g. Choi & Johnson, 2015) that addressed 
the positive effects that instructor-recorded videos explaining content have on students as they 
improve students’ understanding and engagement with the materials. Relevant literature (Kim et 
al., 2019) suggests that audio representation of the content helped make curricula more accessible 
to students with low vision. Therefore, the findings of this study showed the need for inclusion of 
audio discussion modality and instructor’s audio representation of the content to reduce the 
challenge and enhance the learning of students with low vision. 


7. Recommendation, Implication and Conclusion 


The findings of this study serve as a foundation for future research on this topic. The 
literature review presents research on universal design for learning (UDL), especially as it relates 
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to accessibility and assistive technology for students with low vision. This study will help 
instructors and online designers who might teach online courses or might want to improve online 
courses. This study could primarily impact those students with low vision take online courses and 
face challenges, so they will have a better learning experience in online courses. Literature (e.g., 
Barnard-Brak & Sulak, 2010) found that students with invisible disabilities often are not 
comfortable disclosing their disabilities. The findings of this study also indicate online instructors 
should provide audio response to accommodate students with low vision in online courses. 

Based on the interpretations of the findings, this study focused on one type of visual 
impairment; however, a much broader future study can include other visual impairment types. In 
addition, this case study focused on the fully online courses, so future research can be replicated 
in blended courses, including face-to-face and online sessions of similar size and student 
population. The UDL theoretical framework can help future researchers replicate the study by 
focusing on specific principles of universal designs for learning (UDL) because such factors affect 
students with low vision engagement in online courses. Other research can expand the case study 
to examine students’ and/or professors’ perceptions of the accommodations and assistive 
technology for engaging students with low vision in online courses. In addition, this study was 
limited to students with low vision; therefore, future studies can expand this case study to examine 
instructors’ challenges when providing helpful accommodations for students with disabilities in 
online courses. 


The findings for this study showed students with low vision identified a screen reader as the 
most helpful assistive technology in online courses. Future studies can employ a quantitative 
approach to compare two groups of students with low vision to examine the effectiveness of using 
specific assistive technology in online courses. In addition, this study’s findings included the 
students with low vision preference for information delivery methods; future studies can employ 
quantitative methods to compare groups of students receiving different information delivery 
methods in online courses to understand the relationship between information delivery method and 
learning performance. 


The overall purpose of this qualitative case study was to explore the most helpful 
accessibility design and assistive technology for students with low vision in online courses. 
Individual interviews were conducted to obtain in-depth data. This study found that the most 
helpful aspects for online content accessibility for students with low vision are headings, color 
contrasting, and alternative formats for materials, such as Word documents, Rich Text formats 
(RTF) or Adobe accessible PDF files. Overall, this study reveals that online courses require the 
inclusion of more accommodations and better implementation of UDL principles to meet the needs 
of students with low vision. The consideration of the findings of this study may bring about 
significant understanding and renovation in the online courses design that will guarantee equal 
learning opportunities for students with low vision. Online designers, instructors and disability 
resource centers may benefit from this study as the findings can guide their decisions on providing 
support to students with low vision. 
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Appendix A 
Semi-Structured Guiding Interview 


When were you diagnosed with low-vision? 

In which educational level you did recognize you needed more support and 
accommodations from the school or teachers/instructors? 

Do you have other family members who have the same or a similar condition? 

Do you learn from them? Or did you teach them how to deal with low vision in academic 
setting? 


As a student with low-vision, do you prefer online or face-to-face courses? 


Face-to-face course 
Online course 
Hybrid/Blended course 
All types of courses 


a. Why do you prefer that type of course? 


What are the information delivery methods (text such as pdf or word document, audio, 
video) that you find to be most beneficial with regard to your learning in the past online 
courses? 
a. How did you use those methods of delivering information to help your personal 
learning preferences? 
b. What currently unavailable methods of providing the information would you like 
to become available in the future? 


How has accessibility and assistive technology helped you to overcome challenges in 
your online courses? What were these challenges? 


Which types of accommodations and assistive technologies could contribute better to 
your engagement, participation and learning of the content of your online courses? 


According to your experience, what accessibility accommodations in online courses were 
helpful to you and how were they helpful ( to navigate the online courses, to better 
understand online instruction and/or to complete online activities?? 


a. What accessibility design did you need in online courses but did not get to help 
you understand the material? 
. What kind of visual information were you looking for in online courses? 
c. What kind of auditory information were you looking for in online courses? 


What is your wish list regarding accessibility accommodations you’d prefer in an online 


course? Can you describe a specific experience where you felt like you didn’t have access 
to services or accommodations that you thought would be helpful in your education? 
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7. Is there anything else you would like the researchers to know about your online course 
experience regarding your low-vision? 


8. What question should I have asked, but didn’t? 
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1. INTRODUCTION 
1.1 Lesson Study 


A traditional teacher-training method in Japan called Lesson Study is composed of 
the following four steps: goal setting, planning, research lesson, and reflection (Lewis et 
al. 2006). Stigler and Hiebert’s (1999) book The Teaching Gap: Best Ideas from the 
World's Teachers for Improving Education in the Classroom brought Lesson Study 
global attention, describing findings from video recordings of Japanese, German, and 
American eighth-grade classes that were part of the TIMSS(Trends in International 
Mathematics and Science Study) 1995 survey. The book attributed the high performance 
of Japanese junior high school students to the skills of their teachers, suggesting that 
Lesson Study is an approach that provides teachers with such skills. Lewis (2016) states 
that Lesson Study includes many initiatives that are not part of American teaching culture, 
one of which is the importance of “research lessons that connect teachers’ discussions 
and practices.” There is a tacit understanding that classroom improvement in the United 
States ends when teachers receive knowledge; this is not the case with the practice-based 
lesson improvement system seen in the Japanese Lesson Study. Therefore, the research 
lesson process connects discussions about teachers’ empirical knowledge with their 
practice. 

Much research on the Lesson Study method has reported on the nature of research 
lessons. Rock and Willson (2005) stated that a Lesson Study method in which university 
faculty members intervened in the creation of research lessons facilitated the transfer of 
teachers’ learning to their daily work. Lesson Study includes three types of support for 
teachers: social, emotional, and instructional (Laura et al. 2019), suggesting the need for 
support for the creation of research lessons. However, few studies have been conducted 
on models of Lesson Study that include expert support. 
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1.2 Teacher Training in Japan 


The National Institute for Educational Policy Research (2014) found that although 
there is a high demand for teacher training in Japan, there are several obstacles to 
teachers’ participation in such training, one of which is the time consumed by their daily 
work. Japanese teachers have busy schedules because they spend more time on lesson 
planning and preparation, extracurricular activities, and administrative work than teachers 
in other OECD (Organisation for Economic Co-operation and Development) countries. 
This shows that teacher training in Japan needs to be more efficient in terms of time. 
However, little research has examined the Lesson Study model with a focus on time 
efficiency. Therefore, video-based training, which emphasizes time efficiency, was the 
focus of this study. 

A previous study of pre-service teachers indicated that there was an increase in 
teachers’ knowledge when training was provided with clear objectives (Seidel et al. 
2013). In addition, teacher training using portable devices has been shown to be more 
time-efficient and effective than face-to-face training, as it is easier to participate in such 
training (Watanabe & Akahori 2008). This suggests that the video-viewing model is 
effective in teacher training; however, as mentioned earlier, it is important in Lesson 
Study that teachers hold discussions with each other (Lewis 2016). Thus, it was deemed 
necessary to emphasize face-to-face discussions and provide time for face-to-face 
meetings. Therefore, we paid attention to the teaching methods of the flipped class. 

In the flipped class method, basic materials such as explanatory lectures are given as 
homework before class, and the learning necessary to develop applied skills is conducted 
in class (Bergmann & Sams 2012). Flipped training has been reported to promote training 
transfer in the context of enterprise education (Nakahara et al. 2018). It has been pointed 
out that teachers’ attitudes and teaching methods tend to be caught up in the methods they 
have experienced as a reproduction of teacher culture (Kawamura 2002), and the flipped 
training program is expected to provide an opportunity for participants to come into 
contact with Active Learning methods themselves. However, the effects and challenges 
of flipped classroom research are unclear. 


1.3 Training Evaluation 


Describing the need for training evaluation, Kirkpatrick and Kirkpatrick (2006) 
stated: “Evaluation can tell us how to improve future programs.” The evaluation in 
training has four levels: reaction, learning, behavior, and results (Table 1). Kirkpatrick’s 
four-level model is considered to be the most commonly recognized model of training 
evaluation. However, some studies have identified a correlation with transfer training, 
which assesses whether the training is applied in practice, and Levels 3 and 4 (Alan & 
Lisa 2012). 

Therefore, it is necessary to use Level 3 or higher for training transfer. However, few 
studies have evaluated Lesson Study using Kirkpatrick’s four-level model. 
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Table 1. Kirkpatrick’s Four-Level Model 


Levels Evaluation Concept 
1. Reaction What were the participants’ reactions to the education? 
2. Learning What knowledge, skills, and attitudes were changed? 
3. Behavior What change in job behavior occurred due to the training 
program? 
4. Result What effect has education had on the organization and its goals? 


Source: Kirkpatrick and Kirkpatrick (2006) 
2. Purpose 


The purpose of this study was to develop a lesson study model and an evaluation 
framework to support the creation of research lessons that are time efficient. 


3. Method 
The Lesson Study was designed based on the ADDIE model. 
3.1 Survey and Determination of Training Content 
To determine the subject matter of the designed Lesson Study, 42 junior and senior 
high school teachers and 13 teachers with administrative experience were given seven 
training courses: instructional methodology, on-campus branch manager, subject, student 
guidance method, information morality (personal information management, internet, etc.), 


ICT(Unformation and Communication Technology), and school administration. The 
results are shown in Table 2 and indicate a need for instructional methodology. 


Table 2. Results of the In-School Training Questionnaire 


: . Management In-Service Teacher 
Questionnaire M SD M SD 
1 Instructional methodology 4.86 0.35 4.31 1.01 
2 On-campus branch manager 3.73 0.62 3.76 Laz 
3 Subject 4.33 0.94 4.48 0.88 
4 Student guidance method 4.10 0.54 4.05 1.11 
5 Information morality (personal 


information management, Internet, etc.) a02 wee 2 bee 
6 ICT 4.00 0.71 4.26 1.02 
7 School administration 3.80 0.75 3.24 1.34 
In-service teacher (n = 42); management experience teacher (n = 13) 


Bonwell and Eison (1991) defined active learning as a learning method that fulfills 


the following five criteria, as opposed to active learning, which has long been vaguely 
defined: 


* Students are involved in more than listening. 


214 


* Less emphasis is placed on transmitting information and more on developing students’ 
skills. 


* Students are involved in higher-order thinking (analysis, synthesis, evaluation). 
- Students are engaged in activities (e.g., reading, discussing, writing). 


* Greater emphasis is placed on students’ exploration of their own attitudes and values. 


Although active learning was originally considered to be a teaching method in higher 
education, it is defined as proactive, interactive, and deep learning in Japan, and it is a 
teaching method that is also needed in secondary education (MEXT(Ministry of 
Education, Culture, Sports, Science and Technology) 2019). In addition, the participants 
of Lesson Study requested the addition of self-efficacy. Thus, the purpose of Lesson 
Study is “proactive, interactive, and deep learning and instructional design to enhance 
self-usefulness.” 


3.2 Design of Training and Evaluation Methods 


A training course was designed for the above content. As mentioned in the 
introduction to the background, a Lesson Study model was designed employing a flipped 
Lesson Study that does not put pressure on work time. In the Test-Operate-Test-Exit 
model, trainees first select necessary knowledge and skills in a pre-test (Test). Next, they 
learn only the part they did not understand in the previous test (Operate), taking a second 
test to decide whether they have met the criteria (Test). Upon passing, they can complete 
the contents (Exit). Otherwise, they return to the knowledge injection step (Operate). The 
evaluation framework is based on Kirkpatrick's four-levels model. The purpose of this 
study was to assess the extent of training transfer up to level 3. The evaluation framework 
is shown in Table 3, and the designed Lesson Study model is shown in Figure 1. 


Table 3. Evaluation Framework 

Reaction Comparison of content and ease of learning with conventional classroom 
research 

Learning Extent to which knowledge has been increased by watching videos 

Behavior To what extent did you apply the content of the video to your classroom 
practice? 

Efficiency The degree of improvement in the time and quality of research lesson 
creation compared to the conventional lesson study 
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Activities of the Trainer 


@ Selection of Viewing Content 

@ Description of future activities 
Pre Learning survey 

the State of Lesson Study 


®@ confirm ation of progress 


1.Confirm ation of 
knowledge( Test) 


2. Video viewing 
(Operate) 


Activities of the Teacher 


"Understanding of edearning 
*Participation in etearning 
"Group discussion on 
Research Lesson plans 


y fi 


@ Lesson plan improvement 


3.Lesson plan 
improvement 
(Operate) 


@ Post Learning survey 


@ Observation of Research 
Lessons 


@ Summaries of the post-review 
meeting 

@ Related books and literature 

@ Learning and Behavior survey 


@ 5s Actwites n Traning. 
@ is survey 


_. 
4.Evaluation 
(Test) 


5.Lesson Study 
meeting (Exit) 


6. Follow-up 
(Exit) 


: 


e-learning 

*Video viewing 

"Lesson plan improvement 
‘Evaluation 


Fail the Evaluation. 
— (2) or (3) 

Pass the Evaluation. 
— (5) 


Lesson Study meeting 


Discussions on Research 
Lessons based on the 
knowledge learned in e- 
learning 


Figure 1. Flipped Lesson Study Model 


In (1), a group training session is held, and a pre-test is given to all Lesson Study 


participants to select the knowledge required. In addition, the outline of the video content 
and the procedure for viewing the video content was communicated. Next, in (2), we ask 
participants to watch the selected videos and make assumptions. In (3), teachers are asked 
to use the video content in their lesson plans in preparation for the Research Lesson. In 
(4), the teachers who conduct the research lesson are evaluated (3), and those who do not 
conduct the research lesson will be tested on the video. From the results, we return to (2) 
or (3) if necessary. The students must repeat the above procedure, and when they reach 
the required level, they are considered to be ready for the lesson and wait until the 
Research Lesson. In (5), we will hold a Research Lesson and discuss the improvements in 
the Research Lesson at a post-meeting. Finally, (6) concludes the Lesson Study by 
providing comprehensive feedback and insights for development. 


3.3 Development of Training Contents 


We developed video contents for e-learning in inversion training. Ten videos on 
instructional design and three videos on self-efficacy were developed. The video content 
was created by referring to the video content of the remote asynchronous lecture given in 
the class of educational methods and technology, which is a compulsory subject in the 
teaching course at T university in Tokyo. Additionally, narration was added to the video. 
This class covers the basic knowledge of educational technology and has a heavy 
theoretical background. Therefore, we added a scene to introduce a specific teaching 
technique in the active learning video. The videos were delivered on Google Classroom 
so that the participating teachers could view them anywhere, at any time. 


216 


4. Implementation of Lesson Study 


We conducted a developed Lesson Study with the participation of 23 teachers (one 
faculty member conducting the Research Lesson) in A City of Education, Mathematics 
Research Group. The program lasted from July to October 2020. The first workshop was 
held in August to confirm the teachers’ prerequisite knowledge and their ability to 
participate in e-learning. From there, the participating teachers engaged in e-learning 
until the day of the Research Lesson in October. We broadcasted a pre-recorded research 
lesson at a review meeting and conducted observations. Then, the participating teachers 
engaged in e-learning until the day of the research lesson in October. 

In view of the recent social situation, the research lesson was recorded in advance and 
replayed at a reflection meeting for observation. In the reflection meeting, the KJ method, 
which includes individual work with worksheets to ensure the homogeneity of 
perspectives, was applied to the observed lessons. 


5. Data Analysis and Evaluation 


The collected data are currently being analyzed according to the evaluation 
framework. Behavioral analysis will be conducted 1 month after the end of the Lesson 
Study in accordance with previous research on training. One issue is that some teachers 
have watched the videos, while some have not. We plan to qualitatively investigate the 
reasons teachers were not able to watch the videos in the flipped lesson study, according 
to the problems that can be seen from the quantitative data. 


6. Discussion 


We developed and implemented a reversible classroom study to ensure time 
efficiency. Additional research will be conducted to determine which factors are related 
to the presence or absence of video viewing, and whether the model takes into account 
the increasing time constraints of Japanese teachers. 
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Abstract 

In this study, we examined how students made changes in their summaries of 
complex scientific texts, supported by a technology-enhanced feedback system. Referring to 
system-generated indices that describe student mental models of texts, we conducted 
qualitative reviews on selected cases in which students in a graduate-level course wrote at 
least two revisions of the summaries of texts. Learners used feedback to introduce more 
relevant concepts and build an integrated structure in a text by connecting concepts across 
adjacent sentences. 


Introduction 

Summary writing is an instructional strategy that promotes students' deep 
understanding of main ideas and their inter-relations in a text. Also, summaries reveal the 
extent to which learner mental representations have changed as a result of instructional support 
(Graesser, Singer, & Trabasso, 1994; Kintsch, 1988). Accordingly, it is important to understand 
strategies that learners use to improve their summaries when they are provided with 
constructive, extended feedback on their products. To that end, the present study builds on a 
previous study (Kim, McKarthy & Heidari, 2020) that explored various indices used for the 
assessment of students' summaries in surface, structural, and semantic dimensions. Structure- 
related indices demonstrated that students made significant changes in their summary during 
revisions (e.g., the systematic order in which concepts are organized and related in the mental 
model). The present study further examined the nature of linguistic modifications via 
qualitative analysis of selected learner summaries from initial to final versions. The findings 
of the study specified what linguistic changes made in summaries as a result of feedback and 
defined the structural aspects related to quantitive indices. 


Text-Based Analytics 

Text-based approaches to reading comprehension assume that reading or discourse 
comprehension is based on simultaneous activation of processes occurring at three different 
dimensions of surface code, textbase, and situational models. Surface code relates to linguistic 
components at the word and phrase levels, constituting the linguistic content of a text (e.g., 
lexical frequency). Textbase and situational models relate to essential inferences that connect 
ideas from different parts of the text and that connect information from the text to information 
in prior knowledge (e.g., discourse-level cohesion— that is, the degree to which information is 
connected across the text). All these dimensions relate to how students form an organized, 
inclusive, and cohesive mental model of a complex text. 

In a previous study, we used text-based analytics tools such as Tool for the Automatic 
Analysis of Lexical Sophistication (TAALES, Kyle et al., 2018) and the Tool for the Automatic 
Analysis of Cohesion (TAACO, Crossley et al., 2016) that provide indices at the surface, 
structure, and semantic levels, demonstrating the potential of the indices to track changes in 
the three dimensions. For example, Table 1 summarizes structure-related indices. However, it 
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was unclear what structural aspects of learner comprehension were addressed by the indices 
and how students made changes in the structure of their summary in response to feedback 
information. To answer the remaining questions, this study conducted qualitative reviews of 
selected cases. 
Table 1. Structure-Related Indices 

Index Definition 
Overlap N-1S A measure of lexical overlap across sentences and refers to the 
number of sentences whose lemma overlaps with the next two 
sentences in the running text. 
Overlap lemma-2S | A measure of lexical overlap across sentences and refers to the 
number of sentences whose content words (Noun, Adjective, 
Adverb, Verb) overlaps with the next two sentences in the running 
text. 
Overlap CN- 2S A measure of lexical overlap across sentences and refers to the 
number of noun lemma (base of a word) types (unique, non-repeated 
words) that occur at least once in the next sentence. 
Overlap N-2S A measure of lexical overlap across sentences and refers to the 
number of sentences whose lemma noun ( base form of a noun) 
overlaps with the next two sentences in the running text. 
Overlap ARG-2S__| A measure of lexical overlap across sentences and refers to the 
number of sentences whose noun and pronoun lemma (base form) 
overlap with the next two sentences in the running text. 


Data Collection and Analysis 

We used 47 cases in which students in a graduate-level course wrote at least two 
revisions of the summaries of texts in the Student Mental Model Analyzer for Research and 
Teaching system (SMART; Kim, 2019). SMART analyzes students' summaries of readings to 
assess the quality of their summary and then drive formative feedback to help students revise 
their summary. For all the cases, we analyzed the magnitude of changes in the structure-related 
indices and selected three extreme cases (see Table 2): One stayed at a similar level (Student 
4), another showed the most negative change (Student 15), and the other demonstrated the most 
positive change (Student 23). Then, we deployed a case study approach to examine students’ 
different patterns of revisions in response to SMART feedback. We conducted visual 
inspections of selected students’ initial and final versions of the summaries, referring to the 
numerical information of the structure-related indices. 
Table 2. Initial vs. Final Case Data 


Winn ae Overlap Cai Overlap Overlap Overlap Nature of 
mane a N-1S 58 CN- 2S N-2S._ ARG-2S Change 
S4Initial 230 1.00 0.84 0.11 0.53 0.61 Minimal 
S4Final 237 1.00 0.84 0.10 0.61 0.69 
aaa te 0.70 0.50 0.06 0.40 0.40 Nagative 
S15Final 283 0.60 0.42 0.04 0.34 0.34 
oe aoe 1.00 0.70 0.07 0.50 0.50 Positive 
S10Final 296 1.00 0.81 0.10 0.77 0.77 
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Findings and Discussion 

Minimal Change: Student 4 

Student 4 has generated a draft that indicates minimal changes as a result of the 
feedback. The word count for the initial version is 230 words and 237 words for the final 
version. The changes made in the final version include modifications of the content to match 
key concepts in terms of accurate terminology. For example, the initial version reads as “It is 
the responsibility of the program designer to ensure the program has objectives, relevant 
content, expectations and involves leadership” becomes more aligned with the theme of the 
reading as it is changed to “ It is the responsibility of the instructional designer to ensure the 
technology-based learning has impact objectives, application objectives relevant content, 
business impact, expectations and involves leadership.” As to structural changes, the student 
revised his summary, overlapping nouns across adjacent sentences and the next two sentences. 


Negative Change: Student 15 
Student 15's final version shows negative change, referring to structural indices. The 


initial version has a word count of 127 words, and the final version consists of 283 words. We 
expected that the revision work with more words could show improved structure. However, 
consistent with the indices, our qualitative inspection showed new concepts introduced in the 
final version tended to be disconnected from the current concepts. 


Positive Change: Student 10 

We detected notable positive changes from the initial to the final version of the student's 
summary. Student 10 added more key concepts and linked new concepts to existing concepts. 
The final version shows more nouns and pronouns overlap across adjacent sentences and the 
next two sentences. The student might notice that some important concepts and their relations 
were missed, and in response, attempt to include them cohesively during revisions. 


Conclusion 
Qualitative analysis of three cases indicated that learners use feedback to align their 
summaries with the expert model, integrating more relevant key concepts. The most common 
pattern for the development of cohesive and coherent summaries was through the use of 
overlapping nouns and pronouns across adjacent sentences and the next two sentences. 
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Abstract 


To date, while many educators and researchers have greatly argued for the importance of teaching 
computational thinking skills, there is a lack of research on an effective approach to CT education. This study aims 
to examine the effectiveness of two different approaches when teaching computational concepts with block-based 
programming language in an online environment. The study conducted a quantitative research using a pretest- 
posttest randomized-group design. Participants in the study were be 101 undergraduate students enrolled in a 
computing and information technology online course. Two different teaching approaches (inductive vs. deductive) 
were delivered in an experimental web-based instruction during the spring semester of 2020. The knowledge test 
scores revealed that both groups were effective to increase the understanding of programming concepts. The 
inductive group outperformed the deductive group on the programming task of “repeat until”. Although the 
inductive group increased higher in the positive programming attitude than the deductive group, the change did not 
significantly differ between the groups. Lastly, the deductive instructional approach had a more positive attitude 
toward online instruction. Practical implication was further discussed. 

Keywords: computational thinking, programming, online instruction, conditionals, block-based programming 


Introduction 


As programming requires cognitive and non-cognitive competencies, various pedagogical approaches 
should be considered. In a variety of subject areas, attempts to find effective pedagogical approaches have yielded 
significant advances in teaching and learning; however, block-based programming is not a common context in 
previous studies. Also, since block-based programming languages are taught to and played with individuals of all 
ages and at different levels of education, different approaches could be considered. For example, in order for helping 
novice learners increase the process of constructing key programming concepts, research is imperative to find out 
how to teach block-based programming effectively. In particular, conditional is one of the foundational 
computational concepts for controlling logical flow involved in programming and also necessary for solving 


223 


conditional problems. The conditionals usually involve the use of variables and operators to control the algorithmic 
flow, so it would make learners hard to acquire required knowledge and skills in conditionals (Duncan & Bell, 2015; 
Fronza et al, 2017; Grover et al., 2014). To facilitate learners to master the conditional statements of programming 
syntax, there is a need for studies to investigate effective approaches to teach the concepts in block-based 
programming environments. 

Two distinct instructional approaches are divided into “inductive” and “deductive” (Felder & Silverman, 
1988; Prince & Felder, 2006; Rieber & Parmley, 1995). Inductive and deductive teaching is an umbrella term that 
encompasses a range of instructional methods or strategies. The inductive approach proceeds from specific to 
general information. Conversely, the deductive approach goes from general and moves to specific. The two 
approaches have proven to be effective in other subject areas, such as language, mathematics, science, and 
chemistry. However, empirical evidence has not supported a consistent conclusion with regard to the effects of the 
inductive and deductive instructional approaches. Some studies reported inductive approach superior (De Jong, 
Acampo, & Verdonk, 1995; Kersh, 1958; Klauer, 1996; Schelfhout et. al., 2006), while others reported findings in 
favor of deductive approach (Barrish, 1970; Erlam, 2003; Mohammaed et al., 2008; Owen, 2009). The other studies 
found the two approaches equally effective (Forgus & Schwartz, 1957; Shaffer, 1989). More importantly, no other 
studies exist examining the difference between inductive and deductive approach to teach block-based 
programming. Although deductive approach has been found typical to constructing new knowledge in other subjects 
(e.g., Atta, Ayaz, & Nawaz, 2015; Chiapetta et al, 1998; Prince & Felder, 2006; Murawska & Zollman, 2015; 
Shaffer, 1989; Singh & Yadav, 2017; Taha, 2014), inductive approach has not been applied to teach block-based 
programming. This study therefore focuses on how two different instructional approaches would help learners' 
cognitive and noncognitive aspects of programming in block-based environments. 


Purpose of the Study 


The main purpose of this study is to compare inductive teaching with deductive teaching approaches on 
students’ Computational Thinking (CT) knowledge achievement, programming skill, and attitudes towards CT and 
programming. Although the deductive teaching and learning has been found normal to constructing new knowledge 
in programming courses, the effect might depend on different context of learning and/or domain of learning; hence, 
inductive approach requires investigation. The inductive approach proceeds from specific to general information, 
whereas the deductive approach goes from general and moves to specific. Both approaches can offer certain 
advantages, but the biggest difference is the logic of reasoning. One way to advance this area of research is to 
compare the two different teaching approaches for learning a difficult computational concept, which can offer 
insight into instructional strategies. This study aims to examine the effectiveness of two different approaches when 
teaching undergraduate students about the construction of a computational concept. The conditionals (e.g., if/then, 
if/else, while conditional) are one of the difficult CT concepts. Also, these concepts are heavily related to other 
concepts (e.g., variable and operator) and also involved in constructing algorithms of programming. Therefore, 
meaningful understanding of these concepts by the students determined their CT competencies (i.e., knowledge and 
skill). In addition, this study explored the difference in online learners’ attitudinal perceptions towards the two 
instructional approaches. 


This study investigated various learning outcomes of two teaching approaches (inductive vs. deductive) as 
follows: knowledge achievement, programming skill, and attitudinal perceptions. This study has the following 
research questions: 


(a) RQI: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
knowledge of conditionals? 

(b) RQ2: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
programming skill of conditionals? 

(c) RQ3: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
attitude toward block-based programming? 

(d) RQ4: Does the student’s attitude toward instruction differ between two instructional approaches? 


Theoretical Framework 


Inductive and deductive teaching is an umbrella term that encompasses a range of instructional approaches. 
Inductive approach has the following characteristics: (a) Particular cases to general rules, (b) Concrete instance to 
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abstract rules, (c) Known to unknown, (d) Simple to complex; deductive approach, on the other hand, has the 
opposite (Singh & Yadav, 2017). Some studies reported the inductive approach superior (De Jong, Acampo, & 
Verdonk, 1995; Kersh, 1958, 1958; Klauer, 1996; Schelfhout et al., 2006); others reported findings in favor of the 
deductive approach (Barrish, 1970; Erlam, 2003; Mohammaed et al., 2008; Owen, 2009). 


Method 
The study conducted quantitative research approach, and pretest-posttest control-group randomized-group design. 


Participants 

Participants in the study was 101 undergraduate students enrolled in a computing and information technology online 
course at a large southwestern university during the spring semester of 2020. The participants enrolled from varied 
majors, were of various ages, and were both male and female. 


Course Information 

The course focused on a set of core competencies that shapes the background of computer science and also provided 
students with experience of learning Scratch block-based programming language. Out of 15 learning modules, nine 
modules were related to Scratch programming. In the learning module of conditionals, the two different teaching 
approaches (inductive vs. deductive) were delivered in an experimental web-based instruction. 


Learning Content 

The desired learning outcome was to create decision-making algorithms through the use of three conditionals (i.e., 
if/then, if/else, repeat until). Although the content of the topic studied in each group is the same, the sequence of the 
content was given in the reverse reasoning order to each group. 


Instrumentation 

Two groups’ CT competencies will be assessed from (a) Computational Thinking test (CTt; Roman-Gonzalez, 
2015), (b) Dr. Scratch and Rubric programming artifact analysis (Moreno-Leon, Robles, & Roman-Gonzalez, 2015), 
and (c) Attitude survey (Authors). 


Procedures 

The pre- and post-test controlled group method is an effective design to minimize internal threats on experimental 
validity. The participants took a pre-test and pre-survey before taking the first module with Scratch. One group was 
taught on the topic of conditionals by the WBI with inductive teaching approach, whereas the other group was 
taught by the WBI with deductive teaching approach. The length of the WBI instruction took approximately three 
hours. The participants were randomly assigned to different conditions. They were asked to carefully read 
instruction on the computer screen and then followed the steps. After the online instruction, they took a post-test and 
post-survey. The Scratch artifact was collected in the assignment submission. 


Analysis 

The data was analyzed by T-Test or Analysis of Variance (ANOVA) to analyze the differences among group means 
in a sample. Also, multivariate analysis of variance (MANOVA) was used for comparing multivariate sample 
means. Moreover, repeated measures multivariate analysis of variance (RM MANOVA) was used to determine 
whether there were any differences in multiple dependent variables over times. 


Findings 


Researchers identified and coded significance statements pertaining to patterns and challenges in the coding 
journal responses. Codes were applied and revised as needed by combining and eliminating codes for parsimony. 
The final number of codes were counted for each category—being incremental and iterating, testing and debugging, 
reusing and remixing, abstracting and modularizing. According to Scratch coding journal responses, 
undergraduates’ reaction towards their programming experience considerably differed. The results are presented in 
the order of the research questions. 


RQI1: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
knowledge of conditionals? 
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The result of multivariate test for time by instructional approach interaction was not significant (Wilk’s A = 0.93, 
F(3, 97) = 2.56, p = .06, partial 7? = .07). Instructional approach was significant for the score on the “Repeat until” 
conditional (F(1, 99) = 6.43, p = .01, partial 7” = .06). Deductive teaching group made greater gains in the “Repeat 
until” score as compared to those in the inductive teaching group. 


RQ2: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
programming skills of conditionals? 


The total score was significantly lower for the deductive group (M/ = 11.40, SD = 3.93) than for the inductive group 
(M = 13.10, SD = 2.77) ((99) = -2.42, p = .01, d = 0.50) and the difference was moderate. The score for the 
“Tf/then” conditional was significantly higher for the inductive group (MV = 4.53, SD = 1.14) than for the deductive 
group (M = 3.66, SD = 1.56) (#(99) =-3.21, p < .001). The score for the “If/else” conditional was also significantly 
higher for the inductive group (M = 4.57, SD = 1.06) than for the deductive group (M = 4.02, SD = 1.49), (199) =— 
2.13, p = .04). 


RQ3: Which approach, inductive or deductive, is more effective for enhancing undergraduate students’ 
attitude toward block-based programming? 


Both groups made a positive change in the perceived programming attitude, which were statistically significant (F(., 
100) = 6.94, p = .01, partial 7? = .07). The time effect was significant (F(1, 100) = 6.43, p = .01, partial 7” = .06), but 
the interaction between time and instructional approach was not significant (F(1, 100) = 2.93, p = .09, partial 

= .03). 


RQ4: Does the student’s attitude toward instruction differ between two instructional approaches? 


The total score was significantly higher for the deductive group (= 3.69, SD = 0.84) than for the inductive group 
(M = 3.34, SD = 0.84) (¢ (99) = 2.09, p = .04, d = 0.04). The score for Item 1 (“TI like the online instruction’) was 
significantly higher for the deductive group (M = 3.76, SD = 1.10) than for the inductive group (M = 3.27, SD = 
1.15) (99) = 2.17, p = .03). The score for Item 3 (“My interest in the subjects has been aroused.”) was also 
significantly higher for the deductive group (M = 3.62, SD = 1.11) than for the inductive group (M = 3.06, SD = 
1.24) (499) = 0.18, p = .02). 


Discussion and Implications 


First, deductive approach produced a higher learning effect in terms of knowledge gain about conditionals, mainly 
when complex concepts or large numbers of facts are taught. Deductive approach is more effective for learning new 
conceptual information by making inference efficient to online college students. Second, inductive approach 
produced a higher learning effect in terms of programming skills of conditionals, particularly when guided practice 
is provided. Inductive approach is more effective for learning new programming skills by making generalization 
meaningful for online college students. Third, both the deductive and inductive approach produced a higher learning 
effect in terms of programming attitude. Inductive teaching made a greater positive change in programming attitude, 
although it is not statistically significant. Lastly, deductive approach made a more considerable positive effect in 
instruction attitude. Inductive approach may need more planning and resources to create clear guidelines and 
adequate scaffoldings. 


The findings imply that both instructional approaches have proven to be useful when teaching programming 
competencies; however, both advantages and disadvantages exist between the two approaches. As a more diverse 
body of learners begins to learn block-based programming, educators need to consider the possible effects of 
different instructional approaches and incorporate them into their online programming instructions. For instance, 
appropriate Instructional approach should be adopted depending on learning objective(s), target learners, or level of 
complexity. Also, inductive approach needs to be more prepared to implement, and students’ understanding of 
learning topics should be assessed by a variety of measures. Moreover, tools and resources should be utilized for 
inductive approach to facilitate their learning process. Since both advantages and disadvantages exist between the 
two approaches, CS educators or instructional designers need to consider the possible effects of each instructional 
approach and incorporate them into their programming instructions. Although the study has some limitations, it 
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would offer insight into the direction of instructional methods or strategies for learners to have meaningful learning 
experiences in online programming courses. 
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Introduction 


Online education continues to rise. With the outbreak of the global pandemic, various 
institutions have opted to put their courses online so that faculty members, students and staff 
members could be safe (Hodges et al, 2020; Zhou et al, 2020). With the high number of online 
learners, instructors and course designers may need to establish ways of engaging their students 
in an online course. Numerous practices have been suggested for engaging learners in an online 
learning environment. These strategies proposed center around how learner-content interactions, 
learner-technology interactions and learner-learner interactions can be improved to engage online 
students (Chakraborty & Nafukho, 2014). However, very few studies have investigated if there 
are gender differences in student views of the role of learner-content interactions on their 
engagement in an online course. 

This study focuses on learner-content interactions and how it plays a role in learner 
engagement. The learners rated how the following aspects of course content impacts their 
engagement in an online learning environment; personal interests in the subject matter, narrated 
lecture videos provided as a class resource, interactive assignments presented in the class, critical 
thinking assignments presented in the class and problem-solving assignments presented in the 
class. Researchers examined if male and female perceptions of the role of learner-content 
interactions on their engagement in an online learning environment was different. 


Interactions 


According to Zimmerman (2012), “Interaction plays a critical role in the learning 
process. For online course participants, interaction with the course content (learner-content 
interaction) is especially important because it can contribute to successful learning outcomes and 
course completion” p. 152. With the importance of interactions in a learning environment, the 
challenge for course developers and instructors is determining how to promote this type of 
interaction. According to Xiao (2017), 

Despite the fact that interaction with content is inextricably interwoven with learner— 

learner and learner—instructor interactions in conventional face-to-face, campus-based 
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educational settings, how to promote this type of interaction has always been top on the 

research agenda in the field of general education. (p. 124) 

Learner-content interaction are very important in education because “without it there 
cannot be education, since it is the process of intellectually interacting with content that results in 
changes in the learner's understanding, the learner's perspective, or the cognitive structures of the 
learner's mind (Moore, 1989. p. 2) 

In a study conducted to test “a regression model for student satisfaction involving student 
characteristics (three types of interaction, Internet self-efficacy, and self-regulated learning) and 
class-level predictors (course category and academic program),’ the authors found that, 
‘Learner—instructor interaction and learner—content interaction were significant predictors of 
student satisfaction’ (Kuo et al., 2014). Accordingly, this study will explore how learner-content 
interactions may have an impact on student engagement in an online learning environment. This 
study categorized learner-content interaction in an online learning environment into, personal 
interests in the subject matter, narrated lecture videos provided as a class resource, interactive 
assignments and critical thinking assignments. 


Gender Differences 


There are a number of studies which have been conducted on gender differences in online 
learning environment, for example, (Sullivan, 2001; Wang et al., 2009; Ma & Yuen, 2011; 
Yukselturk & Bulut ,2009; Rovai & Baker, 2005). However, only a few of the many studies 
examined if there were any gender differences in learner perspectives on the role of learner- 
content interactions in an online learning environment. According to Yukselturk & Bulut (2009), 

In the literature, gender based differences in education have been recognized as an 

important focus for research for a long time, especially, since increasing number of 

online female students. When reviewing gender related studies, the effects of this variable 

are inconclusive on student experience in distance education. (p. 13) 

Some studies focused on the gender differences in online courses with regard to learner 
perspectives on online courses in general. For example, according to Sullivan (2001), online 
courses were reported to be of great value to female adult learners with family responsibilities or 
with children. Other studies focused on gender differences in learner performance and 
interactions. For example, in one study, researchers found that, 

women studying online are confident independent learners who may outperform their 

male counterparts. They do not have reduced computer and Internet access compared 

with men, nor are they disinclined to enrol [enroll] on online courses. They attach greater 
value to the pastoral aspect of tutoring and have different interaction styles from men, 

which may have implications for online tutoring support. (Price, 2006, p. 349) 

With regards to learner-learner interactions, some studies focused on how male and 
female students communicated in an in an online learning environment. For example, researchers 
in another study found ‘empirical support for the idea that men and women communicate at 
different levels, perceive community differently, and have differing views of perceived learning 
in an online educational environment’ Rovai & Baker, 2005, p. 42). 

In another study which was conducted to determine if there were gender differences in 
learner use of learning strategies and motivation, the authors found that ‘test anxiety explained 
a significant amount of variance in female students' achievement and two variables (self-efficacy 
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for learning and performance, and task value) explained a significant amount of variance in male 
students' achievement’ (Yukselturk & Bulut, 2009, p.12 ). 

Very limited number of studies focused on gender differences in learner perspectives of 
the role of learner-content interactions on their engagement in an online learning environment. 
This study on the other hand, will bridge this gap in the literature by focusing on gender 
differences in learner perspectives on the role of learner-content interactions on their engagement 
in an online learning environment. 


Engagement 


Student engagement is one of the topics which has been studied in the literature, for 
example (Robinson & Hullinger, 2008; McBrien, Cheng & Jones, 2009; Czerkawski & Lyman, 
2016; Kahn, et al., 2017). Student engagement is defined ‘as the interest and motivation students 
have in their own learning of course content’ (Young & Bruce, 2011, p. 220). This study adopts 
this definition of engagement. 


One factor which influences engagement in an online learning are learner-learner 
interactions which are established in learning communities. According to Young & Bruce 
(2011), there is a close relationship between a classroom community and student engagement. 

Another factor related to engagement in an online learning environment is the content. 
According to one study conducted to determine what students found engaging in an online 
learning environment, students reported that activities which involved applying concepts in 
problem solving or case studies, research papers and labs were engaging (Dixson, 2010). 

With regards to gender differences in student engagement, women who were taking an 
online version of a course were found to be more engaged in the course than the male students in 
the same course (Price, 2006). According to the same authors, the women in the online course 
also liked to learn from other students and were more confident. 

Few studies on engagement in online learning environments focused on gender 
differences on the role of learner-content interactions on student engagement. By linking 
engagement to learner-content interactions in an online learning environment, this study will 
examine if male and female students perceived the role of learner-content interactions on their 
engagement in an online learning environment differently. 


Method 
Participants 


Participants in this study (V=147) were undergraduate students from a land-grant 
university in the southeastern part of the United States. The students were enrolled in four online 
courses and age ranged between 18-34 years. About 72 were males and 75 females. Majority of 
the students were Caucasian (n=115). Asians were the second largest group (n= 12), and African 
Americans was the smallest group (n=10). 
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Instrument and Data Collection 


Researchers conducted an initial study using the ‘Student Perception of Engagement in an 
Online Course’ survey to measure student perception of engagement in an online course. One of 
the questions centered on student views on the role of learner-content interactions on their 
engagement. The survey was adapted from surveys that had been used by other researchers and 
based on the literature. The survey consisted of 12 questions. The first questions gathered 
demographic information such as age and race. 


Analysis and Results 


Before conducting an analysis, reports feature in Qualtrics was used to view the results. The 
results were sorted by gender and a side by side comparison was made to observe differences 
between male and females’ views on the impact of the following learner-content interactions on 
their engagement, 

Personal interests in the subject matter 

Narrated lecture videos are provided as a class resource 
Interactive assignments are presented in the class 
Critical thinking assignments are presented in the class 
Problem solving assignments are presented in the class. 

Figure 1 and 2 shows the observable differences which showed that males thought that 
critical thinking assignments where somewhat engaging (n=26) vs females (n=17). However, a 
high number of females (n=33) thought that narrated lecture videos where highly engaging 
compared to their male counterparts (n=19). 
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Figure 1 
Male Scores on Five Aspects of Learner-Content Interactions 


Not applicable EG 


Slightly engaging 


1 My personal interests in the subject matter. 

@ Narrated lecture videos are provided as a class resource. 
1 Interactive assignments are presented in the class. 

§ Critical thinking assignments are presented in the class. 


© Problem solving assignments are presented in the class. 
;omewhat engaging 


Voderately engaging 


Highly engaging 
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Figure 2 
Female Scores on Five Aspects of Learner-Content Interactions 


Not at all engaging 


Slightly engaging 


1 My personal interests in the subject matter. 

ll Narrated lecture videos are provided as a class resource. 
1 Interactive assignments are presented in the class. 

& Critical thinking assignments are presented in the class. 
!) Problem solving assignments are presented in the class. 


Moderately engaging 


Highly engaging 


Independent samples ¢ tests were conducted to determine if there were gender differences 
with regards to learner perspectives on impact of the following learner-content interactions on 
their engagement, 

Personal interests in the subject matter 

Narrated lecture videos are provided as a class resource 
Interactive assignments are presented in the class 
Critical thinking assignments are presented in the class 
Problem solving assignments are presented in the class. 

To evaluate the hypothesis that there were no significant differences between male and 
female students’ perspectives of the impact of the five elements of learner-content interactions 
on their engagement, independent samples ¢ tests were conducted. See figure 4. 

On the impact of personal interests in the subject matter on student engagement, no 
significant differences were found between males and females, ¢ (145) =-.230, p = 0.818. 

On the impact of narrated lecture videos provided as a class resource on student engagement, 
no significant differences were found between males and females, ¢ (145) = -1.567, p = .119. 

On the impact of interactive assignments presented in the class on student engagement, no 
significant differences were found between males and females, ¢ (145) = -.403, p = .688. 

On the impact of critical thinking assignments presented in the class on student engagement, 
no significant differences were found between males and females, ¢ (145) = -.872, p = .385. 

On the impact of problem-solving assignments presented in the class on student engagement, 
no significant differences were found between males and females, t (145) = -.457, p = .648. 
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Figure 3 
Group Statistics 


Group Statistics 


Std. Std. Error 
Gender N Mean Deviation Mean 


Course Content Male 72 4.43 1.309 -154 
Based on your 
experience with all 
online classes, please 
rate the following 
statements based on 


your level of Female 75 4.48 1.298 | .150— 
engagement: - My 
personal interests in the 
subject matter 


Course Content Male 72 4.60 1.206 -142 
Based on your 

experience with all 

online classes, please 

rate the following 

statements based on 


your level of 
engagement: - Female 75 4.91 1.187 “137 
Narrated lecture videos 

are provided as a class 

resource 


Course Content Male 72 4.25 1.275 -150 
Based on your 

experience with all 

online classes, please 

rate the following 

statements based on 


your level of 
engagement: - Female 75 4.33 1.234 .142 
Interactive assignments 

are presented in the 

class 


Course Content Male 72 4.11 1.181 .139 
Based on your 

experienc with all 

online classes, please 

rate the following 

statements based on 


your level of 
engagement: - Critical Female 75 4.29 1.343 -155 
thinking assignments 

are presented in the 

class 


Course Content Male 72 4.01 1.327 -156 
Based on your 

experience with all 

online classes, please 

rate the following 

statements based on | 1 

your level of Female 75 4.12 1.479 171 
engagement: - Problem 

solving assignments are 

presented in the class 
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Figure 4 
Independent Sample Test 


Levene's Test for Equality of 
Variances t-test for Equality of Means 


95% Confidence Interval of 


Sig. (2- Mean Std. Error the Difference 
F Sig. t df tailed) Difference Difference Lower Upper 


Course Content Equal variances .007 .932 -.230 145 -818 -.049 .215 -.475 .376 
Based on your assumed 

experience with all 

online classes, please 

rate the following 

statements based on 

your level of Equal variances not -.230 144.648 819 -.049 215 -.475 -376 
engagement: ~ My assumed 

personal interests in the 

subject matter 


Course Content Equal variances -000 -987 -1.567 145 -119 -.309 .197 -.700 -081 
Based on your assumed 

experience with all 

online classes, please 

rate the following 

statements based on 

your level of 

engagement: - Equal variances not -1.567 144.533 -119 -.309 .198 -.700 .081 
Narrated lecture videos assumed 

are provided as a class 

resource 


Course Content Equal variances .024 .878 -.403 145 -688 -.083 207 -.492 .326 
Based on your assumed 

experience with all 

online classes, please 

rate the following 

statements based on 

your level of 

engagement: - Equal variances not -.402 144.207 688 ~.083 -207 -.493 .326 
Interactive assignments assumed 

are presented in the 

class 


Course Content Equal variances 4.103 .045 -.872 145 385 -.182 .209 -.595 .231 
Based on your assumed 

experience with all 

online classes, please 

rate the following 

statements based on 

your level of - 

engagement: - Critical Equal variances not -.874 143.913 .383 -.182 -208 -.594 .230 
thinking assignments assumed 

are presented in the 

class 


Course Content Equal variances 1.738 .189 -.457 145 648 -.106 .232 -.565 .353 
based on your assumed 
experience with all 
inline classes, please 
a the following 
statements based on 


your level of Equal variances not -458 144.343 647 -.106 .232 -.564 352 
engagement: - Problem assumed 

solving assignments are 

presented in the class 


Discussion and recommendations 


This study examined the differences between male and female students’ views on the impact 
of five aspects of learner-content interactions on their engagement in an online learning 
environment. These aspects are, 

Personal interests in the subject matter 

Narrated lecture videos are provided as a class resource 
Interactive assignments are presented in the class 
Critical thinking assignments are presented in the class 
Problem solving assignments are presented in the class 

After an analysis, the results showed that there are no significant differences between male 
and female students’ perspectives on the role of the five aspects of learner-content interactions 
on their engagement. 
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However, the results also show that some aspects of the learner-content interactions were 
engaging to the learners. For example, both males and females thought that narrated lectures 
videos where engaging highly engaging. However, we recommend the use of shorter videos as 
per findings from Guo et al., (2014) who reported that shorter videos were highly engaging to 
students. 

Critical thinking assignments and personal interest in the subject where also moderately 
engaging for both male and female students. Critical thinking may involve solving unstructured/ 
open-ended, problems (Mastrian et al., 1999). Critical thinking ‘assumes an inquiry and 
hypothesis based approach to ideas as well as thinking that is open to revision’ (Cavdar & Doe, 
2012, p. 298). Incorporating critical thinking assignments in the class could benefit the learners 
by assisting think critically. 


Recommendation for practice 


The authors recommend the use of short narrated videos in online courses. The literature 
is not conclusive on the exact length of an educational video. Mukuni (2020) recommends that 
videos should be between 3-20 minutes long and that long videos should be segmented. The 
videos used should be related to the topic. 

The authors also recommend the use of critical thinking assignments in an online learning 
environment. The assignments used could involve problem solving and could also be open-ended 
(Mastrian et al., 1999). They could be designed in such a way that they enable learners to inquire 
and approach the problem in a hypothesis-based manner (Cavdar & Doe, 2012). 


Recommendations for further research 


Even though this study found no significant differences between male and female views 
on the role of learner-content interactions on their engagement in an online course, further 
studies need to be done on this topic. The aim of identifying gender differences in online 
learning environments is to ensure that both male and females’ differences are accounted for in 
the design of the course. Given the current global crisis which has pushed k-12 to online 
learning, future studies may need to be done in the k-12 online learning to determine if there are 
gender differences between male and female students’ perspectives on engagement in an online 
learning environment. 
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Giving Virtual Office Hours a Makeover: Improving Instructor Presence 


David J. Mulder 
Dordt University 


Welcome to your online classroom! (Now figure it out!) 


As online learning has grown in prevalence over the past decade (Allen et al., 2016; 
Brown & Green, 2019; 2018), new challenges have emerged for both students and instructors 
alike in navigating this learning environment. Instructors who have previously found success in 
face-to-face learning environments may not automatically see those skills translate into 
successful online teaching (Dunlap & Lowenthal, 2018; Lewis, 2018). Many instructors have 
found the transition from face-to-face teaching to online teaching to be an adventure of discovery 
(McGee, Windes, & Torres, 2017). 

The present study is a case study that tells a story: the story of how I became dissatisfied 
with an aspect of my own online teaching practice, and sought to improve it. Many online 
instructors—even experienced instructors—continue to learn how to conduct their craft by 
discovery and by learning through the examples of other instructors (Dunlap & Lowenthal, 
2018). I will refer to learning by discovery as “the school of hard knocks,” that is, learning by 
trying things, sometimes failing, and revising for the next time around. This might sound similar 
to design-based research (Brown, 1992; McKenney & Reeves, 2012) but the telling of this story 
will play out as a qualitative case study (Creswell, 2013) as we consider how online instructors 
approach discovering how to engage students in learning in the online classroom. 


A brief review of relevant literature 

The Community of Inquiry (Col) model developed by Garrison, Anderson and Archer 
(2000) has become a dominant model for explaining interactions in online courses that lead to 
meaningful educational experiences (Dunlap & Lowenthal, 2018; Swan & Richardson, 2017). 
The Col model describes how cognitive presence, social presence, and teaching presence interact 
in dynamic ways to foster the development of community through interpersonal relationships in 
“the pursuit of meaningful inquiry” (Swan & Richardson, 2017, p. 64). While much investigation 
of online learning has been conducted using this model for a theoretical lens, new research 
endeavors building on this model are beginning to investigate instructor presence as a construct 
(Martin, Wang, & Sadaf, 2018; Oyarzun, Conklin, & Baretto, 2016; Richardson & Lowenthal, 
2017). 

Instructor presence has been described as being composed of three elements: teaching 
presence, instructor immediacy, and instructor social presence (Oyarzun et al., 2016). Teaching 
presence is the way the instructor design and organizes the course, plans materials and learning 
activities, and facilitates and encourages learning (Garrison et al., 2000). Instructor immediacy is 
a way of describing the communication factors that instructors use to decrease perceptions of 
psychological distance between the instructor and the students, which can increase student 
motivation, participation, and achievement (Oyarzun et al., 2016). Instructor social presence is 
the ways the instructor demonstrates they are a real person (Richardson & Lowenthal, 2017). It is 
worth noting that several of these elements are aspects of the Col framework, but the way they 
interact for instructors may take on a different flavor than they do for students. These three 
elements of instructor presence come together in the way the instructor facilitates the online 
course, demonstrating their real care and concern for students and their learning. 
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There many methods for instructors to demonstrate their presence in asynchronous online 
courses. Some common approaches include the use of video-based personal introductions, course 
orientation videos, timely response to student questions, sending announcements to the class, 
being active participants in discussion forums along with students, and providing timely, 
personalized feedback on student assignments (Martin et al., 2018). Many of these strategies 
require some planning ahead to incorporate into the design of an online course. However, even 
simple instructional moves, such as addressing students by name, or including personal stories as 
examples can enhance instructor presence (Richardson & Lowenthal, 2017.) 

Another approach online instructors might consider for boosting their presence is the 
inclusion of synchronous, web-based meetings as a component of asynchronous courses (Martin 
et al., 2018). Results of this approach are mixed, but it may merit consideration in some 
circumstances (Elwood & Brauhn, 2017; Lowenthal, Dunlap, & Snelson, 2017; Watts, 2016). In 
particular, synchronous web-based meetings may be an effective way for online instructors to 
hold office hours for their students at a distance when structured effectively (Lowenthal et al., 
2017). 


Method 

The present study is a case study explaining an intervention used to boost instructor 
presence in an asynchronous online course, and an exploration of the students’ response to this 
intervention. The use of case studies has a long history in the field of education (Creswell, 2013; 
Merriam, 1998; Yazan, 2015), but only more recently have they been used in the field of 
Educational Technology (Mardis, Hoffman, & Rich, 2014; Oliver, 2014). In her explanation of 
the value of case study methodology for educational research, Merriam (1998) suggested, “case 
study research in education is conducted so that specific issues and problems of practice can be 
identified and explained” (p. 34), which is the approach taken in the present study. Further, 
Oliver (2014) suggested that applied case studies might be the most relevant for practitioners in 
the field of educational technology because they “directly address the problems of 
implementation in specific contexts they face when teaching with technologies” (p. 912). This 
might be the sort of research encouraged by Reeves & Reeves (2015) of shifting from research 
focused on “things” towards research focused on “problems” (p. 29). 

Case studies are, by their nature, studies of the particular (Eisner, 1991; Yin, 2014). Due 
to the small sample size—often a single case—a key aspect for developing effective case studies 
is researcher reflexivity (Richardson & St. Pierre, 2005; Stake, 2005). Reflexivity involves 
careful introspection on the part of the researcher, and an awareness of their own positionality 
with regard to the case, which they must make explicit in the reporting (Creswell, 2013, Stake, 
2005). Richardson & St. Pierre (2005) suggested that a key marker of reflexivity is a sense of 
self-awareness and self-exposure in the author’s writing. Merriam (1998) urged case study 
researchers to employ “thick description” in their elaboration of the case (p. 30). Thus, in my 
exposition of this case, I will be intentional about describing the role I played as the instructor, 
while also providing a detailed explanation of the context, the problem, the intervention, and the 
students’ response. 


Case study: Reimagining office hours for “Issues in Education” 
As a qualitative case study, this report describes an educational intervention I employed 


to tackle a problem of practice: improving my presence as an instructor, and thus boosting 
student engagement in online office hours. and inviting students into a learning community. As 
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the story of this case unfolds, I position myself as a participant-researcher (Creswell, 2013), 
explain the context for this research and the participants, elaborate on my approach to 
intervention, share the data I collected from students, and describe the results uncovered. 
Considering generalizability of qualitative research, Eisner (1991) suggested that the coherence 
of the narrative is essential, that it holds together and makes sense. My intent in exploring the 
case is to do just that; the particulars of this case may not generalize to every situation, but as a 
coherent story of an educational intervention, the reader may find it informative for their own 
practice. For structuring this report, I appreciated the models of qualitative case studies 
exemplified by Lewis (2018) and van Root and Zirkle (2016). 


Research context 

I serve as an instructor in Master of Education program at Dordt University. Dordt is a 
private university located in the upper midwestern United States. The M.Ed. program has seen 
rapid growth in the past five years, from approximately 60 students in 2015 to approximately 230 
students in 2020. This program is a fully online program with no on-campus residency 
requirements. There are eight tracks in the program, with the largest numbers of students in the 
School Leadership, Teacher Leadership in Curriculum and Instruction, and Special Education 
tracks. Courses are offered in the Fall, Spring, and Summer terms. The focus course for this 
study was offered in asynchronous format during the Summer term, which is a compressed 
schedule, 8-week course that runs from mid-June through the first week of August. The courses 
in the program are designed in modules that correspond to the eight weeks of the course, and 
students moved through the modules as a group week-by-week. 

Students enroll in the M.Ed. program from across North America, and around the world. 
Approximately 75% of the students in the program are from the United States, with 
approximately 15% from Canada, and approximately 10% hailing from other countries, 
including Indonesia, South Korea, China, Thailand, Nigeria, Libera, and Nicaragua. All students 
in the program take a shared core of four courses, including Issues in Education, the focus course 
for this study. While the program is not structured around a cohort model, small class groups are 
utilized to increase students’ sense of connection and camaraderie with their classmates. Typical 
class sizes in the program are approximately 20 students. 


The problem: Lack of participation in online office hours 
I have been teaching online courses since 2010, and much of my early learning about 


effective online instruction came through the school of hard knocks. I tried things, failed, and 
learned from my missteps. This is not an uncommon experience for many instructors new to the 
online classroom (McGee, Windes, & Torres, 2017). I often emulated things I saw other online 
instructors doing, whether it was actually proving effective or not. One of these teaching moves 
was offering virtual office hours for students taking my online courses. I began including 
synchronous office hours for my online students in 2012. What this involved, initially, was 
including a line on my syllabus listing the hours I would available, and several contact methods 
for setting up a meeting: email or text, through which we would set up a web conference. In my 
first six years of using this approach, I had exactly two students take me up on this—not exactly 
a resounding success! It is embarrassing for me to think this through now, because I was clearly 
not designing these opportunities for connection with my students’ needs in mind. I was 
operating out of my own convenience but feeling good about myself for offering students the 
opportunity to connect. 
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A kind colleague pointed this out to me once during a conversation about online office 
hours. I admitted that it was quite pointless for me to just list available hours if I was not going to 
take any steps to actively reach out to my students in a way that really made me more accessible 
to them. Office hours were not a substantial part of my instructor presence, and I decided that if I 
wanted to take this aspect of my teaching practice seriously, some changes were needed. In 2018, 
I began publishing a link on the main page of my online courses that directed students to a 
standing web conference meeting. I explained that I would be present during the designated 
times in the web conference, and they could drop in as desired. This change in approach did have 
an impact: that year, I had three students actually reach out via that web conference link. While 
still not an exceptional result, at least I felt I was moving the needle a bit. But the problem 
remained: if office hours were to be a valuable part of my course—for both my students, and for 
me as the instructor—a re-imagination was needed. 


The intervention: Shifting from “office hours” to “happy hour” 

It was about this time that I my memory was jogged of an experience I had in grad school 
when a friend mentioned attending “happy hour” at a local establishment. I remembered a 
research study I was a part of while I was taking a course with Patrick Lowenthal at Boise State 
University, and I recalled the engaging way office hours played out in that course. I did a little 
sleuthing and found the article that described his own shifting approach to office hours 
(Lowenthal et al., 2017). I remembered Dr. Lowenthal making the comment in one of our office 
hours meetings that “Office hours have a marketing problem—no one wants to come to ‘office 
hours,’ but no one wants to miss ‘happy hour’” (P. Lowenthal, personal communication, July 12, 
2015). I decided I had nothing to lose by trying this small shift in re-marketing office hours. 
While I was making changes, decided to follow a few other suggestions from Lowenthal et al. 
(2017), including ensuring that office hours were optional, recording the meetings so students 
unable to participate synchronously could still view the video afterward, and adding an 
instructional component to each meeting. 

After reflection, I decided to host an online “happy hour” each week during the 8-week 
summer term of 2019, telling students that “the drinks were on them” but welcoming them to 
participate in optional synchronous meetings. Knowing that I would be teaching four sections of 
Issues in Education that term, I decided I could have multiple “happy hour” sessions, and I 
would not restrict attendance to any particular class section for a given synchronous meeting. All 
students were welcome to choose a synchronous session that would work well for them timing- 
wise. I decided to schedule a morning session, an afternoon session, and two evening sessions 
each week. I planned the meetings to be held on Tuesday afternoon and evening, and Wednesday 
morning and evening. Part of the plan for having meetings at different times of day was to 
accommodate my students located outside of North America, for whom holding just one 
synchronous meeting each week would almost certainly prevent them from ever attending. 

As for the content of the meetings, I decided that I would have the same basic plan for all 
four happy hour meetings for a module, and I would publish a brief agenda ahead of time. The 
general plan for each week’s happy hour meeting included an opening check-in with attendees, a 
content-oriented presentation from me as the instructor, and then an open time of discussion, 
wonderings, and clarification based on students’ questions related to course content and 
assignments. Each happy hour meeting was scheduled to last one hour, with about 20 minutes 
spent on each of the three segments of the agenda. Students would not be required to attend but 
were warmly encouraged to participate synchronously in the learning community. I recorded one 
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of the four weekly meetings for each module, and if students were unable to attend, they were 
encouraged to watch at least the “lesson” segment of the videos which were published online. 


Student participation 

A total of 77 students were enrolled across four sections of the Issues in Education course 
during the summer term of 2019., with 19 or 20 students in each section. These students ranged 
from graduate assistants with no K-12 teaching experience, to novice teachers with 1-3 years of 
experience, to moderately experienced teachers with 4-8 years of experience, to veteran teachers 
with 9 or more years of experience in the classroom. Students in the various sections of the 
course were from four different countries: the United States, Canada, Indonesia, and Liberia. 

By my tally, 45 of the 77 students taking Issues in Education participated in at least one 
happy hour meeting during the summer of 2019. Six students attended happy hour all eight 
weeks of the summer term. Some students always came to the same meeting each week (e.g., 
always attending the Wednesday morning meeting) while others varied in their choice of which 
meeting to attend. Attendance varied depending on the rhythm of the term and the topic of the 
meeting. For instance, in week 4 of the term, when I introduced a major research project that 
would take them several weeks to complete, 44 students joined in happy hour meetings held 
during that week to hear more and get their questions answered. In contrast, the last week of the 
term, only seven students attended happy hour meetings. 


Student response to happy hour 
As part of my normal teaching practice, at the end of each term I invite my graduate 


students to anonymously share about their experiences in the course. I do this by way an 
instructor-created survey of mostly open-ended prompts that address different aspects of the 
course: the course content, the structure of the course, and their impressions of the instructor. In 
the summer of 2019, I added additional questions specifically about happy hour to gauge their 
perceptions of this intervention. I built some internal logic into the survey so participants would 
see customized surveys based on their answers to some of the initial questions. The survey 
questions about happy hour are included as Appendix A. Of the 77 students taking the course, 56 
completed the survey. Of the survey completers, 27 students (48.2%) attended happy hour at 
least once. 13 students (23.2%) did not attend at all, and 16 (28.6%) reported that while they did 
not attend, they watched the videos afterward. 

Students who attended happy hour meetings were largely appreciative of the opportunity 
to connect with classmates and with me as their instructor. Main themes that emerged as positive 
outcomes of participating in happy hour were opportunities for personal interactions with 
classmates, opportunities to get their questions answered, and opportunities to hear more of their 
instructor’s thoughts on the content from the readings and discussions. As for advice they had for 
improving happy hour, no clear patterns emerged, but I gleaned a few helpful suggestions, 
including ideas such as incorporating more stories into my lessons, offering some prompts for 
possible discussion topics ahead of time along with the agenda for the meeting, and considering 
replacing one or more discussion forums with synchronous discussions of the readings. Many 
students commented that they appreciated having multiple options for meeting times, which 
made it possible for them attend. 

The students who reported that they never attended happy hour had several reasons why 
they chose not to join in. Nine said they were too busy to attend. Five suggested that when they 
read the agenda, it did not seem that the meeting would be worth their time. Three reported that 
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because the meetings were optional, they just opted out. Perhaps the best answer to any question 
I asked on this survey was in response to the question, “Is there anything that could have made 
happy hour more appealing, or made it more likely you would attend?” One student who had not 
attended any happy hour meetings replied with just one word: “Margaritas.” 

The students who reported that they watched the videos afterward likewise had varied 
reasons for not attending the synchronous meetings. Eleven students reported that they were too 
busy to attend, however none stated that the meetings did not seem they would be worth their 
time. Seven students reported that the times simply did not work for them, and seven noted that 
because the meetings were optional, they just opted out. When asked about what could make it 
more likely to attend, four students wrote comments expressing some version of “If the meetings 
were required, I would have found a way to make it, but because they were optional, I did not 
attend.” Another student reported, “I found the videos to be easy to use... I oftentimes forgot 
about the happy hour sessions taking place, so I would just watch the video afterwards.” Two 
students expressed gratitude for these recordings, because of their busy summer schedules due to 
coaching or a side job, and the recordings made things more flexible for them to feel they were 
fully participating in all aspects of the course. 


Summary and plans for next steps 

Overall, I was pleased with these results. I felt that my first attempt at re-imagining office 
hours as “happy hour” was largely successful. While there was certainly still room for 
improvement, a large majority of students participated, far more than had ever participated in my 
pre-makeover version of office hours. Further, the feedback they shared with me at the 
conclusion of the course was a healthy combination of affirmation and constructive criticism. 

After this positive outcome to my first attempt, I decided to continue my happy hour 
experiment. I was scheduled to teach Issues in Education again in the summer of 2020, and I 
decided to reduce the number of meetings each week to three: a morning, afternoon, and evening 
meeting each week. This was mainly to help me balance my own teaching rhythm during the 
summer, and the evening meetings were the most sparsely attended in my experience during the 
summer of 2019. I made notes for myself after each week’s meetings during the 2019 session 
about what worked well in the synchronous lesson presentations, and what did not work so well, 
and I used this information to revise the content of several of the lessons. Time will tell what the 
next iteration will look like after summer 2020, but I suspect I will continue to use the “happy 
hour” approach for office hours for the foreseeable future. 


Conclusion 

A major goal of this study was to illustrate ways that instructors might boost their 
presence in asynchronous online courses by incorporating synchronous sessions. In my 
experience teaching Issues in Education in the summer of 2019, this is just what happened. More 
students than ever before—both by percentage of the total as well as in sheer numbers— 
participated in office hours after this intervention was put into place. Students reported 
perceptions of positive benefits for their own learning, and many were actively engaged in first- 
hand connections with their instructor and colleagues alike. Speaking as the instructor, I too was 
greatly encouraged by the opportunities I had to build relationships with students, and to give 
them the opportunity to get to know me as a real person. 

This study is a sort of replication study of the work done by Lowenthal, Dunlop, and 
Snelson (2017) with a few twists in the intervention, and a rather different approach in the 
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reporting. As a qualitative case study, this study certainly has limitations; it will not generalize to 
every teaching situation. This is the story of just one instructor who is a participant-researcher 
taking an active role in this study. That said, I have attempted to position myself reflexively and 
provide a rich and clear description of the context of the research, the intervention, and the 
students’ responses, but it remains for the reader to judge the trustworthiness of this report. It is 
my deep hope that examining this story prods the reader to consider ways that they might 
leverage synchronous web conferences as a means of boosting instructor presence in their own 
online courses. Further, I hope that the investigation of this case might inspire more 
contextualized research in a similar vein in the future. As Reeves and Reeves (2015) encouraged, 
more replication studies and explorations of problems of practice in Educational Technology will 
only strengthen the field. 
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Appendix A: Survey items about happy hour 


I really like the idea of happy hour (office hours) for online courses, and I'm working on refining 
the way I conduct them. Your input on the following is very helpful for me in this! 


Did you participate in happy hour at all? 
a. Yes —at least once. 
b. No 
c. I watched the videos afterward. 


(Decision logic: If students answered “a” they were directed to section A, if “b” to section B, or 
if “c” to section C.) 


A. I'm glad you were able to come to happy hour at least once! A few questions about your 
experience in happy hour... 


What did you find beneficial about happy hour? 
What could make happy hour better? 
Were happy hours scheduled at times that worked well for you? Please explain. 


B. No worries that you didn't attend, but I am hoping to have more students join us for happy 
hour in the future, so your responses to these questions will help me better understand why you 
might have opted out. 


Which of these were reasons for not participating in happy hour? Check any that apply. 
I was too busy with other things to attend 

It did not seem like happy hour would benefit me 

The times happy hour met did not work for my schedule 

It was optional, and I just opted out 

Other (please explain) 


i a ce 


Is there anything that could have made happy hour more appealing, or made it more 
likely you would attend? 


C. I'm sorry you weren't able to attend in person, but I am glad you watched the videos 
afterward! I am hoping to have more students join us for happy hour in the future, so your 
responses to these questions will help me better understand why you might not have been able to 
join in. 


Which of these were reasons for not participating in happy hour when it was offered live? 
Check any that apply. 

a. Iwas too busy with other things to attend 

b. It did not seem like happy hour would benefit me 

c. The times happy hour met did not work for my schedule 
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d. It was optional, and I just opted out 
e. Other (please explain) 


Is there anything that could have made happy hour more appealing, or made it more 
likely you would attend? 
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1. Introduction 
1.1 Instructional Design and Technology in Japanese Teacher-Training Course 


In Japan, the Central Council for Education (2016) considers the new curriculum 
guidelines, which are to take effect in junior high schools and high schools in 2021 and 
2022, respectively, as based on the following three pillars: (1) what students will be able 
to do, (2) what they will be able to learn, and (3) how they will be able to learn it. 
Particularly, the last item implies improvements in giving instructions from an active 
learning point of view. 

In response to these needs, the Ministry of Education, Culture, Sports, Science and 
Technology (MEXT 2017) summarized the skills required for the pre-service teacher- 
training courses of all Japanese universities in a document titled “A Core Curriculum of 
Pre-Service Teacher-Training Course.” 

In this core curriculum, pre-service teacher-training courses are divided into four 
categories. Table 1 shows the categories of the curriculum and specific examples of the 
course at the Tokyo University of Science. For example, “Teaching Methods of 
Mathematics Course” falls under the category of “Subjects and Teaching Methods” and 
“Instructional Design and Technology Course” falls under “Teaching Method for Morals, 
Integrated Studies, Student Counseling, and Educational Consultation” at the Tokyo 
University of Science. The purpose of these subjects is to understand the fundamental 
theories and practices of instructional designs. There is a need to improve and gain 
understanding in instructional design theories in Japanese higher education. 
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Table 1. Categories of the Core Curriculum and Specific Examples of the Course in the 
Tokyo University of Science 


Category Specific Examples of Course 

is3 Specialization Subjects and « Teaching Methods of Mathematics Course 
Teaching Methods « Teaching Methods of Science Course 

> ° Basic Understanding of - Foundation of Pedagogy 
Education - Introduction to the Teaching Profession 


¢ Teaching Methods for Morals, 
Integrated Studies, Student 
Counseling, and Educational 
Consultation 


- Instructional Design and Technology (IDT) 
- Extracurricular Activities and Career 
Guidance 


« Preparation for Teaching Practice 


fap aU anOoANETECHCe ¢ Seminar in Teaching Profession 


1.2. Instructional Design in Lesson Plans 


Katsuaki Suzuki (2005) states that Instructional Design is a model and a research 
field that combines methods to increase the effectiveness, efficiency, and appeal of 
educational activities. For example, Gagné's nine events of instruction and the ARCS 
model is an instructional design theory and a model, respectively, in this research. 

In this research, participants designed lesson plans. A lesson plan is a design 
document that outlines the type of lesson to be taught, background, objective, and main 
contents (Ichikawa 2019). 

In Japan, traditionally, the flow of a lesson has three parts, “Introduction,” “Body,” 
and “Summary.” However, Gagné’s nine events of instruction theory is divided into nine 
events, from “Gaining attention” to “Enhancing retention and transfer.” Considering 
these nine events, lesson planners may design a lesson in more detail (Table 2). 


Table 2. Correspondence Table of a Lesson Flow and Gagné’s Instructional Events 


Lesson Flow Instructional Event 


1 Gaining attention 
Introduction Informing the learner of the objective 


Stimulating recall of prerequisite learned capabilities 


Presenting the stimulus material 


Providing learning guidance 
Body 


Providing feedback about performance correctness 


2 
3 
4 
5 
6  Eliciting performance 
7 
8 Assessing the performance 
9 


Summa: : : 
| Enhancing retention and transfer 
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1.3. Today’s Japanese Students’ Motivation in Secondary Education 


As aresult of Programme for International Student Assessment (OECD 2018), 
Japanese high school students have the top level of mathematics literacy. However, 
according to the result of TIMSS (2015), Japanese junior high school students have low 
motivation toward Mathematics. Therefore, the problem lies in the gap between high 
literacy in mathematics and low motivation to learn. 

To control students' mathematics motivation, we focused on the ARCS Model that 
stands for Attention, Relevance, Confidence, and Satisfaction. The ARCS Model is a 
problem-solving approach for designing motivational aspects of learning environments to 
stimulate and sustain students’ motivation to learn (Keller 1987). 


Table 3. Categories of ARCS Model and its Subcategories 


Category Subcategory 

Attention Perceptual Arousal, Inquiry Arousal, Variability 

Relevance Familiarity, Goal Orientation, Motive Matching 

Confidence Learning Requirement, Success Opportunities, Personal Control 
Satisfaction Natural Consequences, Positive Consequences, Equity 


2. Purpose 


We are not sure if the students can design the lesson plan more effectively through 
the Instructional Design and Technology Course. Do the motivations of Instructional 
Design change? Therefore, the purpose of this study is surveying how learning the 
Instructional Design theories affect the pre-service teacher-training students’ Instructional 
Design and motivation. 


3. Methods 
3.1. Instructional Design and Technology Course in Our Field 


The objective of the Instructional Design and Technology (IDT) course includes 
learning instructional design theories and practical methods. The participants were the 
students of Tokyo University of Science, specialized in Mathematics, Physics, Chemistry, 
and Biology. 

Tokyo University of Science, founded in 1881, is one of the oldest private 
universities of science and technology in Japan. The number of students in this university 
is approximately twenty thousand as of May 2020. They have a pre-service teacher- 
training course for junior high school and high school. The students can get their 
teacher’s license in Mathematics, Science, and Information. The number of teacher’s 
license issuance is about 600 per year on average. Therefore, this course carries great 
importance. 

Furthermore, they have Active Learning Classroom in the Kagurazaka campus, 
Tokyo. In this room, the students can use projectors with Electronic Blackboard. Also, 
desks and chairs are movable to make a group easily. 
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3.2. Flow of Research 


Figure 1 shows the flow of the research that is composed of two Studies. Study 1 was 
held as a face-to-face lesson from September to November 2019. Study 2 was held as an 
online lesson from May to July 2020. In both studies, the participants answered the 
questionnaires and designed lesson plans twice through the course. 


Study 1 Study 2 


September 2019 GRR evicus oases Users May 2020 Answer pre-questionnaire 
Take 1st lesson Take 1st lesson 
Design pre-lesson plans Design pre-lesson plans 


Take 2nd ~ 6th lessons Take 2nd ~ 7th lessons 


Create notebooks for an exam 
Take last lesson (exam 


1DY=*-}(e] alm ofes= 1 (=s=t-10) ae) t=] 81s) 


Create notebooks for an exam 


Design post-lesson plans 


Take last lesson (exam 


November 2019 | Nuseip elesieetiesilen elas July 2020 GRA ctes oleksi ste LSS Cola lare ics 


Figure 1. Flow of Studies 
3.3. Learning Contents of IDT Course 


Table 4 shows the contents of the course. They include fundamental instructional 
design theories and models. 


Table 4. Learning Contents of IDT Course 


Title of Lessons Specific Contents 


¢ Educational Technology, Technological 
Pedagogical Content Knowledge 

¢ ID, Gagné’s five types of Learning Outcome, 
Gagné’s Nine Events of Instruction 


Introduction 


Designing Instructions 


Active Learning ¢ 21“ Century Skills, Active Learning 
ID Considering Students’ Motivation * ADDIE model, ARCS model 
Learning Environment ¢ Learning Space, Activity, and Community 


The Consistency of Objective and 


Aseeuuent Method ¢ Teaching Objective, Way of Assessment 


¢ History of Instructional Media, Practical Use 


Instructional Media of ICT 


252 


3.4. Questionnaire of Studies 1 and 2 


Instructional Design Motivation 


We created 24 items of Instructional Design Motivation for pre-service teacher- 
training course students with reference to a six-level definition of learning experience 
(Parrish and Wilson 2008) using a 5-point Likert scale (1 = I don’t think so and 5 =I 
think so). Each level has four items (e.g., “I am interested in the content of ID” for level 
1). In this research, we use this value in Study 2 to focus on the difference between pre- 
and post-score. 


Teacher Efficacy 


We quoted 7 items related to subject instruction from the teacher efficacy scale for 
the Faculty of Education (Igarashi, Miyauchi, and Yamada 2018) using a 5-point Likert 
scale (1 = I don’t think so and 5 =I think so, e.g., “I can teach in a way that is easy to 
understand”). 


Change of Instructional Design Motivation 


In the post-questionnaire, we asked whether students changed instructional design 
motivation using a 5-point Likert scale (1 = declined and 5 = increased). Subsequently, 
we asked why they answered as they did both from the perspective of their own 
motivations and environmental causes. 


Teaching Experience Level 


We created 1 item of Teaching Experience Level for pre-service teacher-training 
course students with reference to a six-level definition of Learning Experience 
(Kawamoto, Watanabe, and Hidaka 2018) using a 6-point Likert scale ranging from 1| for 
“T am not interested in this class. I do not think this class is useful” to 6 for “ID is 
important in life. I am happy to design instructions assuming the environment of 
education.” 


Self-Regulated Learning Strategy 


We quoted 24 items related to self-regulated learning strategy from the scale of self- 
regulated learning strategy for university students (Hatano, Oikawa, and Hanzawa 2011) 
using a 5-point Likert scale, ranging from 1 for “I don’t think so” to 5 for “I think so.” 
(e.g., “I take a lesson motivated whenever I am not interested in the learning contents.”’). 


3.5. Assessment 
To assess the participants’ lesson plans, we developed checklists for two 


perspectives: Gagné’s Nine Events of Instruction from 0 to 9 points and the ARCS Model 
from 0 to 4 points. 
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In study 2, we provided these checklists to the students so that they could check by 
themselves. 


4. Result 
In total, 141 participants answered both pre- and post-questionnaires and designed 
pre- and post-lesson plans (Table 5). To accomplish our goals, we checked qualitative 


changes in lesson plans and in students’ motivations. 


Table 5. Number of the Participants for Each Specific Course and Study 


Specific Course 


Study Science Total 
Math Physics Chemistry Biology 
1 18 8 Fe 0 4] 
2 63 9 23 5 100 
Total 81 17 38 5 141 


4.1. Result of Lesson Plans 


Gagné’s Nine Events of Instruction 
Table 6 shows the result of a three-way analysis of valiance (ANOVA) of Gagné’s 
Nine Events of Instruction. The three factors in this analysis are “Study (Study 1, Study 


2),” “PrePost (Pre, Post),” and “Course (Mathematics, Science).” 


Table 6. Three-way ANOVA of Gagné’s Nine Events of Instruction 


Factors SS df MS F i 
Study 3.08 1.00 3.08 22d 0.01 
PrePost 41.67 1.00 41.67 29.97 ™ 0.09 
Course 1.90 1.00 1.90 ees 0.00 
Study x PrePost 9.50 1.00 9.50 6.83 0.02 
Study x Course 0.11 1.00 0.11 0.08 0.00 
PrePost < Course 2.10 1.00 2.10 1.51 0.00 
Study =< PrePost x Course 0.34 1.00 0.34 0.24 0.00 
Residuals 380.87 274.00 1.39 

n= 141, SS: Sum of Squares, MS: Mean Square Fe pes O10, *** p< 001 


As a result, there was a significant Study x PrePost interaction. Therefore, we 
conducted Tukey’s Multiple Comparison of Gagné’s Nine Events of Instruction. As a 
result, there were four significant differences. Especially, in Study 2, students’ lesson 
plans improved from the perspective of Gagné’s Nine Events of Instruction, ¢ (274) = 
7.36, p < .001, d= 0.44. 
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Then, we conducted McNemar’s Test for the value of Gagné’s Nine Events of 
Instruction in Study 2 (Table 7). As a result, almost all events improved. However, only 
about 10% of the students could design the summary part. 


Table 7. The number of Accomplishment of Gagné’s Nine Events 


Event nPre “MPs 7(1) 

1 Gaining attention 14 37 23.00 “* 
2 Informing the learner of the objective 40 64 24.00 “™ 
3 Stimulating recall of prerequisite learned capabilities 24 49 23.15 ™ 
4 Presenting the stimulus material 84 86 =: 0.50 

5 Providing learning guidance 1 9 8.00 ™ 
6 Eliciting performance 42 i dae (ry ames 
7 Providing feedback about performance correctness 0 3 a 

8 Assessing the performance | 13. 12.00 ™ 
9 Enhancing retention and transfer 1 9 640 * 
n= 100 * p < .050, *** p < .001 
ARCS Model 


From the perspective of the ARCS Model, we conducted the same test as ANOVA 
(Table 8). As a result, there was a significant Study x PrePost interaction. 


Table 8. Three-way ANOVA of the ARCS Model 


Factors SS df MS F WV? 
Study 2.09 1.00 2.09 327 0.01 
PrePost 10.02 1.00 10.02 15.68.""" 0.05 
Course 1.14 1.00 1.14 1.79 0.01 
Study = PrePost 4.39 1.00 4.39 6.87 ™ 0.02 
Study x Course 0.19 1.00 0.19 0.29 0.00 
PrePost x Course 0.02 1.00 0.02 0.02 0.00 
Study x PrePost x Course 0.01 1.00 0.01 0.02 0.00 
Residuals 174.99 274.00 0.64 

n= 141, SS: Sum of Squares, MS: Mean Square ** n < 010, *** p < .001 


Therefore, we conducted Tukey’s Multiple Comparison of the ARCS Model. As a 
result, there were two significant differences. Especially, students could use the ARCS 
Model for their specific course through the learning style of Study 2 compared to that of 
Study 1, ¢ (274) = 3.13, p< .05, d=0.19. 
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(Table 9). As a result, all perspectives of the ARCS Model improved. However, all 


Then, we conducted McNemar’s Test for the value of the ARCS Model in Study 2 


perspectives remain to be less than 50% of lesson plans. 


Table 8. Number of Accomplishment of the ARCS Model 


Perspective of the ARCS Model nPre ——“MPost 7(1) 
Attention 29 46 152] 
Relevance 6 20 42295 ~ 
Confidence i 34 9300 
Satisfaction 1 17° 14.22-°" 
n= 100 Hy < 001 


4.2. Result of Questionnaires 


Instructional Design Motivations in Study 2. As a result, the students could design 


Table 10 shows the result of Wilcoxon’s Signed Rank Test for the value of 


instructions considering learning content. However, their specific course, such as 
Mathematics, is less useful in designing instructions. 


Table 10. The Change of the Instructional Design Motivation 


Pre 


Post 


More 


SDpre 


Mpost 


SDpost 


Mpost- 
More 


10 


11 


13 


I gain something 
through ID. 

I know why I 
design 
instructions. 

I grow through 
ID. 

The other specific 
courses I take are 
useful to design 
instructions. 

The other teacher- 
training courses I 
take are useful to 
design 
instructions. 

I spend a lot of 
time designing 
instructions in my 
school life. 


3:91 


3.61 


3.56 


3.76 


3.82 


2.82 


0.82 


0.84 


0.83 


0.92 


0.80 


0.90 


4.24 


3.95 


3.95 


3.46 


4.05 


3.19 
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0.73 


0.78 


0.83 


1.10 


0.80 


1.09 


0.33 


0.34 


0.39 


-0.30 


0.23 


0.37 


460.50 


526.50 


aK 


7K 


-0.27 


0.36 


0.42 


Designing 
15 instructions are 3.65:. 0.82 3.91 0.90 0.26 760.50” 0.31 

interesting. 

I try to adopt 

various teaching 
18 methods when I 3.41 082 3.68 0.84 0.27 591.00 * 0.33 
design 
instructions. 
I change my 
teaching methods 
depending on the 
learning contents. 


19 3.30 0.77 3.56 1.01 0.26 889.00 * 0.28 


n= 100, 5-point Likert scale * pn < .050, ** p < .010, *** p< .001 
5. Discussion 
5.1. Discussion on Lesson Plans 


Based on the above results, the positive effects of learning instructional design 
theories are as follows: 
¢ In Study 2, students’ lesson plans improved from the perspective of Gagné’s Nine 
Events of Instruction. 
¢ Students can use the ARCS Model for their specific course through the learning 
style of Study 2. 
¢ Therefore, Providing Checklists of the Instructional Design theories may help 
design instructions. 
On the other hand, there are a few limitations as well. 
¢ Only 3 ~ 13% of the students could design a summary part of the lesson plan. 
¢ All perspectives of the ARCS Model were less than 50% of the students’ post- 
lesson plan. 
Therefore, to ensure whether the students can design a summary correctly, peer 
review or other review systems might be helpful. 


5.2. Discussion on Motivation 


The positive effects on the students’ motivations are as follows: 
¢ By learning Instructional Design Theories, students can know why they design 
instructions. 

¢ Students can design instructions considering learning content. 

Therefore, they may consider instructional design not only as a single task, but also 
as a meaningful task. 

However, there is a limitation in the perspective of students’ specific courses being 
less useful for designing instructions. Therefore, their instructional design might only be 
useful for secondary education but not higher education. 
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Gamification refers to the application of game mechanics to non-game settings in order to 
promote motivation and increase engagement (Brian, 2014; Li, Dong, Untch, & Chasteen, 2013). 
Many tools have been introduced to gamify learning in all kinds of educational settings. Among 
these tools, digital badges are considered as an achievement-based gamification tool (Grouling, 
Hedge, Schweiger, & Cinder, 2014) which have been gradually adopted in higher education for 
about 10 years (Abramovich, 2016; Grouling, et al., 2014). Although some researchers claimed 
using digital badges does not involve real game thinking, which includes competition, 
cooperation, exploration and storytelling (Kapp, 2012), most educators agree that gamification 
can involve using leaderboards, point rewards, achievements, and badges (Grouling, et al., 

2014). 

In educational settings, digital badges are used to deliver instruction and track learning 
outcomes and learning progress (Rimland, & Raish, 2019). Researchers pointed out digital 
badges can support self-regulation (Cucchiara, Giglio, Persico, & Raffaghelli, 2014) and increase 
self-awareness (Abramovich, 2016). Cheng, Waterson, and Newby (2018) stated that digital 
badges can help with goal setting in learning process. Other researchers claimed the 
implementation and use of digital badges not only motivates learners but also provides a new 
method for assessment and creates alternative credentials (Newby, Wright, Besser, & Beese, 
2016). 

Although the existing literature has discussed the advantages of using digital badges as a 
gamification tool, research on the effectiveness of using digital badges in higher education are 
still limited. Therefore, researchers have called for future studies on how to use digital badges 
and what instructional strategies should be used (Abramovich, 2016; Cheng, Waterson, & 
Newby, 2018). Other researchers called for investigations on using gamification tools in different 
contexts (Abramovich, 2016), such as in writing classes (Grouling, et al, 2014). In respond to 
these calls, this case study chose two digital badge tools to gamify a college level writing class at 
a Midwestern university. Through surveys and interviews, the study will investigate the 
effectiveness of gamification, student perception of gamification, and the impact of gamification 
on student learning behavior in the writing class. 


Background and Purpose of the Study 
The issues and problems of college writing class has been discussed throughout the last 


two decades. Poor preparation from high school is one of the biggest issues of college writing 
class, especially in first year class (Hughes, 2009). Some educators claimed students are less 
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engaged in writing class than other subjects that directly related to their majors (Tantawi, Sadaf, 
& Alhumaid, 2018). Therefore, correcting negative behaviors and increasing engagement and 
collaboration should be considered in today’s college writing class (Grouling, et al, 2014). On 
the other hand, traditional lectures and PowerPiont slides are not enough for today’s writing class 
(Coffin, Curry, Goodman, & Hewings, 2003; Grouling, et al, 2014), and new strategies and tools 
need to be adopted in teaching academic writing to accommodate millennial students (Coffin, et 
al, 2003). Other researchers addressed using technology in academic writing classes has multiple 
advantages, such as improving students’ responsibility for their own work (Levine, Ferenz, & 
Reves, 1999) and encouraging peer responses (Jin, 2007). 

Among these technologies, gamifications and digital badges gained increasing attention. 
Grouling, et al (2014) reported using digital badges in a college writing class improved perceived 
enjoyment of collaboration and student self-reported attendance. Similar results were reported 
form multiple subject areas in other universities/institutions (Abramovich, 2016; Rimland, & 
Raish, 2019). Some universities even developed their own digital badge platforms to embrace 
gamification in teaching and learning, such as University of Central Florida (Cheng, et al, 2018), 
UC Davis, and Purdue University (Newby, et al., 2016). Purdue University reported more than 
10,000 students has been using Purdue self-designed digital badge tools, and these tools become 
one of the most commonly used educational tools on campus. To improve the effectiveness of 
college writing class and explore potential gamification ways using digital badges, this case 
study investigated student perceptions, learning behavior, and the effectiveness of these 
gamification tools in a college-level academic writing class. 


Research Design 


An undergraduate-level first-year composition class at a Midwestern university was 
chosen for this case study. The course was delivered face-to-face in the spring semester of 2019. 
The academic writing course emphasizes writing as a process, and writing projects assigned to 
students throughout the semester traditionally consist of five papers, each with multiple drafts, 
and only the final drafts are graded. Following this assessment and instructional approach, and in 
line with the literature, the instructor realized the challenge in engaging students in writing drafts 
for each assignment and in class collaboration. In order to improve teaching, the instructor 
incorporated two university self-designed digital badge tools to gamify the course. 

Specifically, one of the tools was used to demonstrate students’ competencies and record 
their achievements with badges. A total of 18 badges were linked to the five assignments and the 
corresponding drafts. Students received a badge once they submitted one part of the assignment 
on time and met all the requirements. Students who received all of the 18 badges were awarded 
extra 20 points on their final course grade. In addition, students can choose to show the earned 
badges to their peers or make it private. 

The other tool was used to deliver gamified online quizzes in class to improve 
collaboration. To take the quizzes, the class was divided into multiple groups. Each group were 
asked to access to this tool and choose an avatar before taking the quiz. The whole group 
received a badge once they completed a quiz. The instructor showed the quiz results leaderboard 
on the big screen. The top 5 scored groups’ avatar were displayed on the leaderboard. A total of 
3 online quizzes were designed for this class. 
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The class contained 19 students aged from 18 to 20. All of the participants were trained to 
use these tools at the beginning of the semester. Technology support was provided when the 
course was delivered. 


Research Questions 
This case study will be conceptualized to answer the following two questions: 
1. How do students perceive the effectiveness of gamification tools in the writing class? 
2. Do gamification tools improve collaboration between students? 
3. Is there any relationship between student perception of gamification and their academic 
performances? 


Data Collection and Analysis 


This study collected two types of data. One is the number of badges and the course final 
grades that each of the participants earned, which is collected by the instructor. The other type of 
data comes from the online survey and face-to-face interviews, collected by the first author. 

Before the end of the semester, the participants were invited to take an online survey that 
evaluated student perceptions of the two gamifications tools respectively. The survey included 
two sets of 5-point Likert scale with 13 statements. As a cross-check, they were also invited to 
participate in an interview which included 10 questions about the overall perceptions of 
gamification tools and the learning strategies that were used in the class. To gain better insight 
into the effectiveness of the gamification tools in instruction, we also interviewed the instructor 
at the end of the semester. 


Research Question 1 

A total of 18 students took the online survey. The responses of the statements were 
summed up to get the overall perceptions of the two gamifications tools. The points that each 
statement received ranged from | to 5. The average points that each statement received ranged 
from 2.58 to 3.5. Table 1 shows the descriptive statistics of the survey results. 
Table 1 
Descriptive Statistics of Online Survey 


Statements Min Max Mean Median Mode 
Q1.1 Earning badges makes me enjoy academic 1 5 238 - 2 2 
writing. 

Q1.2 Earning badges makes academic writing 1 4 258 G25 4 
easier. 

Q1.3 Earning badges helps me complete my 1 e) 3.33: «4 4 
assignment on time. 

Q1.4 Earning badges helps me understand 1 5 3.33 4 4 
assignment requirements. 

Q1.5 Earning badges helps me to value writing asa_ 1 1 215° 3 1 
process. 

Q1.6 Earning badges ensure efficiency of peer 1 4 241 2.5 1 
editing. 

Q1.7 The digital badge tool is easy to use. 1 5 3 4 4 
Q1.8 I enjoy using the digital badge tool. 1 5 Zio” 23 cS) 
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Q2.1 Taking quizzes using the gamified online 1 4 6) GAS 4 
quizzing tool is fun 


Q2.2 Taking quizzes facilitates in my learning of 1 4 SAT 3:3 4 
academic writing concepts 

Q2.3 Taking quizzes using the gamified online 1 5 3.33 4 4 
quizzing tool improves my class participation 

Q2.4 The gamified online quizzing tool is easy to 1 5 3.25 4 4 
use. 

Q2.5 I enjoy using the gamified online quizzing 1 5 a2) 3 3 
tool. 


The top ranked statements for the digital badge tool are: “earning badges helps me complete my 
assignment on time”, “earning badges helps me understand assignment requirements”, and “the 
digital badge tool is easy to use”. The top ranked statements for the gamified online quizzing tool 
are: “taking quizzes using this tool improves my class participation’, “the gamified online 
quizzing tool is easy to use”, and “I enjoy using the gamified online quizzing tool”. The 
following screenshot image shows the badges on the instructor’s dashboard. Student dashboard 


ges they earned. 


(0) Passport 
Group Awardees 

GROUP MENU 

Home Name Group Badges Awarded 

Badges 18 al 
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INSTRUCTOR TOOLS @ a!) @ 
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Edit Group x @ 
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Figure 1. Instructor Digital Badges Dashboard 
(Note: students’ names are deleted from the image) 


The interviews focused on the students’ specific experience of gamification tools. A total 
of 12 students participated the interview. Overall, the students gave positive feedbacks on the 
gamification tools and the class activities that were designed with the tools. Students claimed 
digital badges helped them track their work and remind them to finish the assignments on time. 
Such comments include: “the digital badge tool is a checklist for me. It’s very helpful to remind 
me submit my assignments on time”, “The digital badge tool keep me more organized. The listed 
due dates make it easier’, “Digital badge is helpful because it monitored my progress”, and “This 
tool helps me track what I did and reminded me to finish the assignments”. 

When asked about their opinions on the activities that are designed using the online 
quizzing tool, more than half of the participants mentioned using the gamified tool is a fun way 
to take quizzes. One students mentioned “we only used this tool three times in the class. I wish 
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we used it more.” When asked if they would like to use these two tools again in the future, all of 
the interviewees said yes except for one who commented “not sure”. Most students are expecting 
to use these tools in other classes as well. 

On the other hand, some students offered negative feedback on these tools. One said the 
tools were very confusing and hard to use. The other one thought the tools didn’t help with 
his/her study at all. One students claimed that submitting assignments through a digital badge 
tool creates confusions because other courses use learning management system to collect 
assignment. 

Research Question 2 

Research question 2 is specific to the online quizzing tool. The whole class was divided 
into several groups to take the online quiz in class. The top 5 groups showed up on the 
leaderboard of the tool. At the end of the quiz, the instructor showed the leaderboard on the big 
screen in class. Both of the survey responses and the interviews results were used to answer this 
question. 

As aforementioned, in the Likert scale, the statement “Taking quizzes using the gamified 
quizzing tool improves my class participation” is among the highest ranked statement. Based on 
the interview, more than half of the participants mentioned that the tool made group work 
interesting. One student mentioned “using this tool get everyone involved in the team work’. The 
other students claimed “I communicated more with other members when using this tool”. Some 
students claimed that group work was helpful and this tools should be use more often. An 
interesting finding from the interview is that almost 1/3 of the students claimed they would like 
to use this tool individually rather than using it for group work 
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Seahorse 


Level 2 
petete 
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Citation 4/1/19 4:50 PM sea] 


© View Learner Achievements 
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Figure 2 Quiz 2 leaderboard. 


Research Question 3 

To answer research question 3, students’ final grades and the online survey results are 
analyzed. For the final grades, the total points that students received ranged from 589 to 1000. 
About 88% of the participants received more than 80% in final grades. One students received 
60%, and one student received 58%. 

Correlation test didn’t show linear relationships between students overall perception of 
gamification tools and their final grades, which is likely because of the small sample size. 
Another possible explanation is that the final grades also included students’ attendance and 
presentations, so they did not perfectly represent students’ writing performances. 
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We found 5 out of 18 students received all the 18 badges and were awarded 20 points to 
add on their final grades. Based on the survey results, these 5 students gave very positive 
comments on the tool. One of the students stated: “I like to use the digital badge tool. It keeps me 
from been lazy. I'm motivated to do the quality work instead of getting extra credit.” The other 
students claimed “I like the idea to earn each badges when I work on the essay. Helped me catch 
up. I like the gamified online quizzes. It was good to test knowledge. Taking the quiz in group is 
also helpful”. 

An interesting finding is that the three students gave lowest ratings on the online survey 
statements actually received 88-90% in final grades. Two of them did not attend the interview. 
The student attended the interview gave almost opposite feedback on the effectiveness of the 
gamification tools that were used in class. The student mentioned “the digital badge tool helps 
the teacher organize the materials and it motivated me in a better way.” The students also 
claimed “using the digital badge tool is better than nothing. It make the class easier.” 


Findings and Discussions 


This case study looked into the impact of gamification tools in a college level writing 
class for the purpose of finding an innovative way to improve this class. Student perceptions, 
achievement, and learning behavior were analyzed. Our findings can be concluded into the 
following five aspects: 

First, overall, 67% participants agreed that earning badges helped them understand 
assignment requirements and complete assignment on time. About 60% students agreed that the 
gamified quizzes improved their class participation and facilitated learning. However, the survey 
results also shows students perceptions on the gamification tools were highly varied. The rating 
range for each of the survey statement is from 1 to 5. Based on the interview, this is likely 
because students had different technology experience. 

Second, the perceptions and effectiveness of the gamification tools are impacted by the 
teaching strategies that are designed by the instructor. Table 1 shows students have more positive 
perceptions on the gamified online quizzing tool over the digital badge tool. This is likely 
because the quizzing tool not only provided digital badges and leaderboard, but also because it 
was used in group works. Class discussions and collaborations were greatly improved by using 
this tool. 

Third, student learning behavior impacted their opinion on using the gamification tools. 
More than half of the participants claimed they enjoyed the group work. They also addressed the 
gamified online quizzing tool made the group work fun. The instructor observed increased class 
discussions and collaborations. This verified the previous studies on improving class 
collaboration (Grouling, et al, 2014). On the other hand, a few students claimed they would like 
to use the tool individually instead of in groups because they prefer to work by themselves. In 
regard of this issue, we suggest future study focus on engaging students with different learning 
behaviors. 

Fourth, from the instructor’s perspective, using digital badges motivated the instructor to 
redesign the course in a better way. The instructor of this class pointed out that using the 
gamification tools helped her rethink the course structure, update the instructions for each of the 
assignment, revise assessment plan, create class activities, and create new rubrics to help 
students understand the requirements of the assignments. The instructor also claimed using the 
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gamified online quiz tool created fun times in class and greatly helped her engage students in 
class. 

Last, using gamification tools and learning management system at the same time could 
cause confusing. Although students were trained to use the tools at the beginning of the course, 
they were expecting a simpler way to earn the badges and take the gamified online quizzes. For 
the instructor, tracking students’ achievement across systems took a longer time than focusing on 
one system. For this reason, we call on future studies in bridging the LMS and the gamification 
tools, especially for fully online classes. 
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Abstract 


29 


This paper presents an action research study that used a flipped approach for “one-shot 
library instruction. Lack of time for delivering effective library instruction is a common issue 
academic librarians face when invited to provide one-shot sessions for students. This study 
compared student’s responses across three levels of “flipped” instruction, as well as examining 
student reactions to the library instruction experience. Animated video lectures were used for 
instruction prior to in person sessions with the librarian. Exercises in the live session provided 
practice for students on searching for articles as well as data on students searching behaviors. 
Surveys after the session gave additional data about the flipped and in person instruction. 
Students with greater exposure to the videos showed a higher degree of assignment completion. 
Results were viewed as a helpful basis for structuring and developing library instruction. 


Keywords: flipped classroom, library instruction; action research 


General Background 


In conventional introductory college or university library instruction, librarians are 
invited to class to present an overview of library services and resources. By working with the 
class instructor, librarians customize the content based on class needs. Instruction topics can 
include describing and showing overall library services as well as basic database searches. 
Distinguishing scholarly work from trade publications, and explaining the use of library 
resources for literature reviews are other topics. 


As with other academic institutions in higher education, library instruction services Kent 
State University strives to educate students on finding and using information effectively and 
ethically. That instruction involves not only the skills and critical thinking associated with 
Information Literacy, but also the skills and reasoning related to technology and media fluency. 
Each librarian is assigned to a limited set of colleges and disciplinary departments across the 
university, and a goal is to support student learning appropriate for where students are in their 
understanding and skill levels. Traditionally, in-class library instruction has involved lectures or 
“show and tell” type presentations. Students have been given a lot of information and librarians 
have regularly been invited back to provide instruction. Overall, the comments received from 
students and instructors in the past have been positive at Kent State. A common comment has 
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been “the session was useful and very informative”, and records indicate that more than 3,400 
undergraduate students were provided in-class library instruction during 2019 at Kent State. 


“One-Shot” Instruction Sessions 


Although the usual presentation type of instruction has been well received, there have 
been concerns with ““one-shot”” library instruction approaches (Artman et al., 2010; Mery et al., 
2012). It is challenging when librarians are expected to pack all the essentials of information 
literacy into a 45-60-minute class time limit, and often these instruction opportunities for a 
course are one time only. Also, timing is important; students learn what they care about and 
remember what they understand (Ericksen, 1984). So, if the session is not offered when students 
feel a need, the effectiveness of library instruction sessions is challenged. Based on our 
experience with traditional instruction, as librarians and instructional designers we sought to 
address concerns about “one-shot” presentations. 


Action Research and Active Learning 


We saw action research (Efron & Ravid, 2020) as a means to understand our current 
instruction and to work toward improvements. We also hoped to incorporate more active 
learning approaches to move the student learning experience beyond just (passively) receiving 
information in the traditional one-shot session (LeMire et al., 2019; Wegener, 2018). Learning 
by doing would be a guiding principle. 


The traditional lecture-centered instruction is a content-oriented approach, which mainly 
involves verbal information delivery. This delivery can use instructor presentation, videotape 
demonstration, class discussion, and textbooks and worksheets as the primarily curriculum 
material (Mastropieri et al., 1999; Scruggs & Mastropieri, 1993). Activity-oriented approaches 
promote a depth of understanding through learning process that emphasizes hands-on experience, 
(Mastropieri, Scruggs, & Magnusen, 1999; Scruggs & Mastropieri, 1993). In response, for this 
study, instead of only presenting information (i.e. lecture, demo), we sought to incorporate 
structured instruction involving in-class hands-on practice and project-based Q&A for the in- 
class session. 


Flipped Classrooms 


Additionally, for this study we focused on a flipped classroom approach (Minuti et al., 
2018; Murray et al., 2015). These approaches can be viewed generally as involving work by 
students outside of class (frequently before class) which is reinforced with activities in class 
(Bergmann & Sams, 2012). Recent studies have indicated that using flipped approaches can have 
value for library instruction, including more motivated students (Karabatak & Polat, 2019) and 
having students come to class better prepared to engage and experience higher level learning 
(Loo et al., 2016). Other work indicated that while challenging, flipped approaches could be used 
for information literacy instruction (Rodriguez, 2016). Additionally, in their background review 
of the literature, a recent meta-analysis also noted that flipped approaches have become widely 
utilized, supporting both active and self-regulated learning, both potentially positive for students 
(Strelan et al., 2020). While some have concluded that a flipped approach did not improve 
learning outcomes (Miller, 2017), to us it seemed overall that the literature suggested that a 
flipped approach could be a good strategy to meet challenges of “one-shot” instruction. 
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To briefly summarize, given our understanding of active learning and flipped approaches, 
the current authors sought to pursue the study described below to help determine the efficacy of 
such approaches for library instruction at Kent State. Results would help guide planning for 
future library instruction development. Specifically, the current study had two objectives. First, 
we sought to examine the use of a flipped approach with students in a psychology writing class 
by comparing students’ responses across three levels of “flipped” settings. A second objective 
was to examine student experiences with how the librarian flipped the classroom. 


Research Design 


An undergraduate psychology research writing course was chosen for this study! because 
the subject librarian had an existing relationship with the psychology department and was invited 
to give library instruction for the course. 


Students and Classroom Setting 


There were three instructors for the writing course involved in the current study, and, 
across the three classes, a total of 149 undergraduate students were enrolled in the course 
received library instruction (Class A: 50 students, Class B: 50 students, and Class C: 49 
students.) All library sessions took place in a classroom with desktop computers located in the 
library. The different teaching styles of the three instructors resulted in the varied levels of 
students’ preparation for the in-class library instruction session. 


Instruction Provided Before In-Class Instruction 


Five library tutorial videos were provided in advance, so students would have 
opportunities for instruction prior to coming to the library session. The videos on search strategy 
covered topics including choosing keywords from a topic, developing search strategy, using 
databases, and accessing full text articles. The instructor in class A incorporated our pre-class 
materials into her course plan by designating time for students to watch videos together, while 
the other two instructors in classes B and C were more “laid back”. Those two instructors 
mentioned the pre-class materials for library instruction session to students. However, they did 
not require students to watch the videos. 


In-Class Library Instruction 


Each of the three classes had two in-class exercises and the same total class time (45 
minutes). The in-class library session had two components: in-class exercises and librarian-led 
instruction. The librarian-led instruction involved a varied amounts of “hands-on” activity. As 
shown in Figure 1., the librarian-led instruction was provided between two exercises. The 
exercises served as a pre-test (Exercise 1) and post-test (Exercise 2) from data collection 
standpoint. The type of librarian-led instruction for the class C was similar to the past, which 
consisted of standard lecture and demonstration, while there was more time devoted to hands-on 
practice in Classes A and B. 


1 The Kent State University Institutional Review Board reviewed and gave the required approval for our study. 
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In-Class Searching Exercises 


Exercise | (pre-test) was used at the beginning of the library session, and Exercise 2 
(post-test) was used after the library instruction. Each exercise involved searches for scholarly 
articles. Students were asked to find a full-text article on a topic given to them, and to complete 
a worksheet as follows. In response to a psychology topic given to them, they searched for a full 
text article, and record the selected keywords and the article title they found on the Exercise 1 
and 2 worksheets, respectively. Additionally, on Exercise | worksheet students were asked to 
record the number of videos they had watch prior to the in-class library instruction session. Each 
of the keywords that students recorded on the worksheets was counted as one-point, with a 
maximum of two points (at least two keywords were recorded), and a point minimum of zero 
(i.e., no keyword was selected). After coding, the collected data were analyzed through 
descriptive statistics (e.g., means, frequency, and percentage) to represent the participants’ 
performance on the article search activity as well as their experience of the library instruction 
session. All student participation and recorded responses in this study were voluntary and 
anonymous, including surveys and worksheets. 


Survey After the In-Class Instruction Session 


Also, following the in-class sessions, a brief survey was used with students. A 4-point 
Likert Scale question was used to assess students’ overall experience in the instruction session, 
and a 10-point rating scale question was employed to assess participants’ perspective on the 
usefulness of library instruction content. Additionally, a follow-up communication was used with 
the instructors regarding the student assignment that involved searching full-text scholarly 
articles on the “testing effect” (Testing Effect, n.d.). 
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Class A (n=50) Class B (n=50) Class C (n=49) 
(Before 
Video- Required Video- Voluntary Video- Voluntary Library 
Peed os eee tera ie hese ee ne ee ere a session) _ _ 
Exercise 1 Exercise 1 Exercise 1 
(5-10 minutes) (5-10 minutes) (5-10 minutes) 
| | | (During 
se sat Library 
Activity-focused Activity-focused Lecture-focused session) 
Instruction Instruction Instruction 
(25 minutes) (25 minutes) (25 minutes) 
Exercise 2 Exercise 2 Exercise 2 
(5-10 minutes) (5-10 minutes) (5-10 minutes) 
* Interview class ¢ Interview class ¢ Interview class 
instructor instructor instructor (After Library 
* Distribute post- * Distribute post- * Distribute post- session) 
session survey session survey session survey 


Figure 1. Research Process Flow 


Results & Discussion 


A total of useable 116 worksheets were collected from all three classes: 40 from class A, 
35 from class B, and 41 from class C, for a response rate of 77.8%. One reason for the missing 
worksheets may be that students did not return their worksheets to the librarian. Forty 
worksheets (80% response rate) were collected from Class A, thirty-five worksheets (70% 
response rate) were collected from Class B, and 41 worksheets (84% response rate) were 
collected from Class C. We looked at the number of pre-class videos watched, in-class exercise 
worksheets, and the assignment completion rate for overall student performance. We also looked 
at students’ response/feedback about their overall experience. There were 55 students who 
responded to the survey invitations: 23 from A, 10 from B, and 22 from class C. 


Results showed that students who watched more videos before the library session had 
higher assignment submission completion rate. As shown in Table 1, Class A had the highest 
average number of videos watched (4.85 out of 5 videos), because the instructor designated time 
for group watching. Importantly, that class also had a 100% assignment completion rate, which 
was very positively noted by the instructor. The assignment completion data were different for 
class B (1 late submission) and class C (5 missing submissions). The average number of videos 
watched and exercise scores were also lower than class A (average 1.97 videos watched by class 
B, and 0.32 videos watched by class C.) 
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Table 1 


Videos watched, mean scores exercise I and 2, & assignment completion rates 


per vee cae eee Assignment submission 
(total =5) (total=2) (total=2) 
(n=O) 4.85 183 tO ieee aiemictcl) 
— jf © & Ce 
(a=4l) 032 176 EM ernibeip abate) 


Table 2 
Overall experience with the library session 


Disappointing As Expected Better than Much better than 


Expected Expected 
A (n=23) 0 4 11 8 
B (n=10) 0 0 3 7 
C (n=22) 0 3 11 8 
All (n=55) 0 7 25 23 
Table 3 


Overall usefulness of the library instruction (scale 1-10, 1= least useful) 


Gs A B C All 
a (n=23) (n=10) (n=22) (N=55) 
Mean 9.0 8.37 8.36 8.48 


The results for keywords selection showed that the majority of students recorded at least 
one keyword from the given topic; the average keywords recorded for Exercise | (pre-test) was 
1.79, and the average keywords recorded for Exercise 2 (post-test) was 1.62. Looking closer at 
the results for each class, results showed that for Exercise 1, Class A students recorded the most 
keywords (1.83) from the given topic, followed by Class B (1.8) and Class C students (1.76). A 
similar range of results was seen for Exercise 2; Class A students had the highest mean of 
keywords recorded (1.95), followed by Class B students (1.57) and Class C (1.34). Interestingly, 
the results are in line with the average number of videos watched before the instruction session, 
and the assignment completion rate. A possible reason for that pattern is that students who 
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watched more library videos developed a better understanding of the research article search, 
including selecting proper keywords for conducting the article search. However, with the current 
data we have, that possible effect of watching videos would need additional confirmation. 


Fifty-five of the 116 students who attended the library instruction session completed the 
online survey. The response rate was 47.8%. Based on survey responses, overall the students 
considered the library session helpful and better than expected, which suggested a positive 
experience (see Table 2 & 3). When examining students’ overall experience with the library 
session, results showed that 41.8% of the participants indicated their experiences were much 
better than expected, 45.5% of the students reported better than expected experience, and 12.7% 
of the students reported their experience was as expected. None of the participants reported 
disappointing experiences. 


Conclusion 


For our current action research study, we implemented a flipped approach with a one-shot 
library instruction session. And we came away from our study with a modestly enhanced view 
that combining flipped and active learning approaches for library instruction is valuable, and 
may be preferred to the “one-shot” instruction we have traditionally developed and used. Of 
course, students’ perceptions and affective factors are important for learning, and the results of 
our pilot study suggest that overall the student’s experience with this new approach was positive. 
Additionally, the positive assignment completion rate results might reflect a possible combined 
impact of repeated exposure to searching techniques and practice. Students involved in “more 
active elements” (class A) did better in on their assignments. Instructor comments indicated 
appreciation for using videos prior to the library session. Varied and repeated exposure to and 
practice with content seemed useful. Going forward, we will be recommending the kind of 
combination we used. That is, to provide students with learning materials that cover key concepts 
prior to in-class instruction so some in-class time can be used for more hands-on activities that 
assume and build on what the students are provided prior to the in-class instruction. 
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Abstract 


Performing effective library database or internet searches are key skills for 
student academic success. It can be challenging for students when working in unfamiliar 
domains, and psychology may be unfamiliar for many students. Even with some 
psychology coursework completed, ineffective approaches, such as “Just Google it”, can 
be common for students when searching for articles. They can end up scrolling through 
hundreds of hits or simply picking a few from the top. This pilot study analyzed 39 
screen recordings of searches conducted by students for an in-class search activity at the 
beginning of a library instruction session for their psychology writing class. Most 
students completed the activity within 3 minutes and reported they found a full-text 
article on the worksheet. Some also utilized advanced features provided by the databases, 
such as the Boolean operator “AND” and limiters (e.g., publication year, peer-review, 
etc.). However, evaluating the worksheets showed that only about half of students found 
a full-text article. Additionally, utilization of advance search features did not necessarily 
lead to better search outcomes, possibly due to poor choice or misuse of keywords while 
searching databases. Our data will help with planning for more relevant and effective 
library instruction. 


Keywords: information seeking behaviors; video analysis; action research 
Introduction 


Performing effective library database or internet searches are key skills for 
student academic success. However, it can be challenging for students when working in 
unfamiliar domains, and psychology may be unfamiliar for many students. However, 
even with some psychology coursework completed, “just Google it” can be a common 
approach to searching for information for students. The general goal of our current study 
was to assess student searching. 


Although research on student search strategies has been pursued for years, this 
kind of studies continues. For example, significant work has recently looked at student 
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use of federated search resources (Dalal, Kimura, & Hofmann, 2015; Dempsey & 
Valenti, 2016), and others have looked at discovery layers and research databases 
(Evelhoch, 2018). Related work has compared student searching on Google and federated 
search resources (Asher, Duke, & Wilson, 2013; Georgas, 2013, 2014, 2015), while 
another research has looked solely at Google Scholar (Schwieder, 2016). 


As is common in many university settings, a regular service at the authors’ 
institution involves librarians providing instruction for writing intensive courses. A goal 
expressed on a sample course syllabus states that one learning outcome of the course is 
that the student will be able to conduct “...literature searches using online research 
databases to find and access topic-relevant published research.” 


Of course, a significant goal is to make sure that students use databases specific to 
a discipline; and PsycInfo is a top choice for topics in psychology; PsycInfo is a database 
for scholarly research in psychology and related disciplines, produced by American 
Psychological Association (American Psychological Association, n.d.-a). Also, a key 
strategy in searching has been and continues to be Boolean searching, although when to 
focus instruction on that may be changing (Lowe et al., 2018). Another basic goal for 
student success in searching is the accessing of full text (Evelhoch, 2018). 


Our general goal has been to assess student’ learning needs and to develop library 
instruction with effective content and teaching strategies. To that end, the purpose of the 
current qualitative study was to explore undergraduate student's online searching 
behavior, and we focused on a common class task, finding full-text scholarly articles for 
research assignments. The primary research question was this, “how do students conduct 
searching to find full-text articles using the Internet?” Also, and more specifically, “What 
sources and search strategies did student use to find full-text articles on a given topic?” 
The findings from this kind of study can indicate for us what students know, giving us a 
basis for designing and developing instruction that is built on the existing skill sets of 
students. 


Method 


Exploratory research can facilitate the description of or insight into a situation or 
an area of social life; it has been employed in research aiming to deepen understanding of 
a phenomenon (Edgar & Manz, 2017; Stebbins, 2001). In the present study, screen 
recordings of in-class search activities and search activity worksheets were used to see 
how students search, to gain a better understanding of students' knowledge and ability to 
effectively search for research-oriented information. 


Participants 


Participants in the current study were students enrolled in a psychology writing 
intensive class at a public research university in the United States. The students, second 
year students, had also either completed a research methods course in psychology or were 
currently enrolled in that course. 
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Data Collection 


Data were gathered from screen recordings of students' hands-on search activities 
during the library instruction session in a computer lab located in the library. Prior to 
delivering the library instruction, students were asked to complete a hands-on search 
activity, which involved students searching for a full-text scholarly article on the topic 
given to them and taking steps they would normally use. The students were also asked to 
write down the title of article they found. After the library instruction, students were 
asked to do a similar exercise again with a different topic. Both worksheets were 
collected at the end of the library instruction session and the computer screen of 
searching process was recorded. 


Data Analysis 


Students’ searching behavior patterns in the screen recordings were analyzed with 
open coding and descriptive statistics to gain insights and identify patterns of student 
searching behaviors. Open coding refers to identifying concepts based on its dimensions, 
this was used for establishing overarching themes (Merriam, 2002; Corbin & Strauss, 
2008). The recurring behavior patterns identified in the screening recordings were viewed 
as themes and subthemes. The coding process facilitated a deeper understanding of the 
behavior patterns being studied. Coding was used for the following factors: 1) time spent 
on the activity, 2) number of sources used for their search, 3) the sources they used, 4) 
whether or not they found a full-text article and 5) search strategy used. The data was 
coded by two independent coders to ensure the validity and reliability. The process also 
ensured that the emerged themes corresponded to and informed answers to our research 
questions. 


Results 


A total of 39 screen recordings of search activity were collected and analyzed. 
Results showed that the average time the 39 students spent on this search activity was 2 
minutes 40 seconds, the maximum time was 7 minutes 27 seconds, and the minimum 
time was 24 seconds. The majority of students (84.6%) used one source to conduct 
searches for this activity, four students (10.3%) used two sources, and two students 
(5.1%) used three sources. The results showing students completing the hands-on search 
activity quickly and using one source suggested that the search activity was easy for the 
most of students. 


Looking at the sources that students used for searching, results showed that the 
sources most used was Discover at Kent State (DKS), accounting for 61.7% of searches. 
DKS is the university’s library search tool conducting searches across multiple library 
databases. Alternative sources used included Google Scholar (19.1%), Google (6.4%), 
PsycInfo (6.4%), and other sources (6.4%). Google Scholar identifies scholarly literature 
in many subjects and sources such as books, peer-reviewed papers, abstracts, and reports 
(Google Scholar, n.d.). As noted before, PsycInfo is a database for scholarly research in 
psychology and related disciplines. Two related sources used for searching were PsycNet 
and Psychology and Behavioral Sciences Collection. PsycNet is a platform designed for 


276 


searching APA databases (American Psychological Association, n.d.-b). Psychology and 
Behavioral Sciences Collection is an EBSCO database that includes full-text scholarly 
journal articles of psychology and behavioral sciences related subject areas, such as 
psychiatry and psychology, anthropology, and behavioral and mental process 
(Psychology and Behavioral Sciences Collection, n.d). 


Table | 
Search Behavior Characteristics (N=39) 


n % 
Number of sources used 
1 33 84.6 
2 4 10.3 
3 2 5.1 
Full-text found 
Yes 19 48.7 
No 20 51.3 
Advanced search feature used 
Yes 13 33.3 
No 26 66.7 


Importantly, 19 of the 39 screen recordings displayed a webpage with a full-text 
research article, which suggested that 19 students found a full-text research article. On 
the other hand, 20 recordings displayed no webpage with a full-text scholarly article, 
which suggested the 20 students “thought” they found a full-text scholarly article or at 
least they reported that they found one on the worksheet. 


With respect to the search strategies, all the 39 screen recordings showed that 
typing a search phrase in a search box was used for conducting searches. In addition to 
typing a search phrase in a search box, thirteen of the 39 recordings (33.3%) showed that 
advanced search features were applied, while 26 of participants (66.7%) did not use 
advanced search features. More discussion of advanced search strategy data is given 
below. 


The above results provide an overview of our general themes. When we looked 
further, more on search behavior patterns emerged, and the following sections present 
more detail. In addition, it will be seen that some similarities and differences in search 
behavior patterns emerged between students who found a full-text article and those did 
not. 


Advanced search features 


Results showed that among the 13 people who used advanced search features, 11 
students used one advanced search feature to conduct a single search, and two students 
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used two features. The advanced search features used included the Boolean operators 
AND, OR, and NOT (46.7%), special characters such as asterisk mark for truncation 
(46.7%), and search filters (6.7%). The search filters used were one or more of the 
following to limit the search results: 1) full-text, 2) scholarly journals, 3) academic 
journals, and 4) language English. 


Table 2 
Advanced Search Feature Used and Full-Text Found (N=39) 


Advanced search feature used 


Yes No 
Full-text found n % n % 
Yes 5 38.5 14 53.8 
No 8 61.5 12 46.2 
Total 13 100 26 100 


A number of patterns emerged on the relation between advanced search feature 
use and whether or not a full-text article was found. For example, among the 13 students 
who used advanced search features, eight (61.5%) did not find a full-text article, and five 
found it (38.5%). On the other hand, among the 26 students who did not apply advanced 
search features, 14 (53.8%) found a full-text article, and 12 did not find it (46.2%). 
Interesting patterns were also seen in the use of search filters used and finding full-text 
scholarly articles. Among the eight students (8) who used advanced search filters but did 
not find a full-text article, six students used Boolean (AND), and one (1) used search 
filters, and one (1) student used/tried both Boolean logic and search filters. Moreover, 
among the five participants who applied advanced search features and found a full-text 
article, four students tried search filters and one student used both search filters and 
special search characters. Although using advanced search features properly to conduct 
an effective search seems important, the results suggested that using advanced search 
features does not necessarily lead to finding a full-text article. 


Search strategy 


As mentioned above, the use of search phrases in the search box search box was 
the strategy applied by all the participants. However, different behavior patterns emerged 
from the way that search phrases were keyed in the search box. Discrete behavior 
patterns showed how students use keyword searches for searching academic full-text 
articles. As we viewed and analyzed screen recordings, we found and used four coding 
patterns of search phrases: 

1) Typing in the whole sentence in one search box, 

2) Typing multiple keywords in one search box, 

3) Typing multiple keywords in multiple search boxes, and 

4) Typing single keyword in one search box. 


Typing in the whole sentence in one search box, the first category, is considered a 
relatively inefficient search strategy when using academic databases. Next, typing 
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multiple keywords in one search box to search for scholarly full-text articles is regarded 
slightly better but similar to the first category. People who apply this type of search 
strategy may indicate not knowing how to use advanced search strategies. Typing 
multiple keywords in multiple search columns generally leads to more precise search 
results when using academic databases. 


The coded data (see Table 3) showed that during the searching process, typing 
multiple keywords in one search box was the most common search method (34.6%), 
followed by typing a sentence in one search box (32.7%), typing multiple keywords in 
multiple search columns (28.8%), and typing a single keyword in one search box (3.8%). 


Table 3 
Search Strategy Used in the Whole Search Process 


Frequency % 
Whole sentence in one box 17 S2ul 
Multiple keywords in multiple boxes 18 34.6 
Multiple keywords in one box le) 28.8 
Single keyword in one box 2 3.8 


When we further analyzed the search strategy that the 39 participants used on 
their first attempt (see Table 4), we noticed that typing multiple keywords in one search 
box (14 students), and typing a sentence in one search box (14 students) were the two 
common methods students applied to begin their search. Nine of the 39 students typed 
multiple keywords in multiple search box and two students used a single keyword. The 
results suggested that most participants conducted searches with inefficient strategies. 


Table 4 
Search Strategy Used at the First Search Attempt (N=39) 


n % 
Whole sentence in one box 14 509. 
Multiple keywords in multiple boxes 14 35.9 
Multiple keywords in one box 9 2351 
Single keyword in one box 2 Sl 


Some other interesting patterns were seen as we compared the time spent on 
search behavior completion for students who found a full-text research article and for 
those who did not. Results showed among students who did not find a full-text article, the 
average time they spent on the task was 2 minutes 27 seconds, the range of time from 24 
seconds to 6 minutes 26 seconds. In comparison, the average time for students who found 
a full-text article was 2 minutes 54 seconds. The maximum time spent searching was 7 
minutes 27 seconds, and the minimum time spending was 36 seconds. 
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Discussion 


The purpose of our research was to qualitatively explore the online searching 
steps used by undergraduate students when they need to find full-text scholarly articles 
for research assignments. We wanted to get a sense of student knowledge of (and skill set 
for using) library sources and tools before delivery of this particular in-class library 
instruction for a writing intensive course in psychology. 


Although time consuming, analyzing screen recordings provided valuable 
information and potentially useful insights regarding undergraduate students' knowledge 
and abilities for conducting scholarly article searches for course assignments and research 
projects. 


The coding themes that emerged gave us a picture of participants’ search 
behaviors. Those results were helpful for addressing our intended goal for of this hands- 
on activity. Interestingly, the university library’s search tool, DKS, was used by most of 
students to conduct their scholarly article search. This is interesting; most of these 
students in a psychology research writing class, who had other psychology course 
instruction, who were asked to find a psychology research article, did not use PsycInfo. 
One likely explanation for that result is the convenience of accessing DKS. The 
university library’s homepage is the default setting for all the web browser software 
installed in the library’s computers, and DKS is prominently located on the library's 
homepage. Future studies should consider altering the default setting of web browser 
from university library’s homepage to other webpage to reduce potential bias. At the 
same time, as librarians we perhaps should not assume that students are inclined or able 
to use PsycInfo on topics in psychology, even after one or two courses in psychology. 


In addition to initial search resource choices, other search patterns allowed us to 
see what students know, and where understanding of strategies for effective database 
searches might be improved. For example, we found that most students (28 students) 
typed the whole search topic given to them or almost the whole topic in one search box. 
Two possible explanations for our results are: 1) these students do not know how to pick 
keywords for their search, and 2) some students may not know how to conduct efficient 
search with multiple search boxes (Boolean logic) as found in the EBSCO advanced 
search interface for PsycInfo. 


We also were interested in what students might do after not finding anything or 
after not being satisfied with their search result. Our results showed after the first attempt 
fails, five of 14 students who use multiple keywords in one search box changed their 
strategies to multiple terms in several search boxes. They did move some of the keywords 
into other search boxes. More research is needed to learn more about students’ thinking 
process or decision making when conducting an internet search for scholarly articles. 


In addition to ineffective search strategies, an interesting finding deals with full 


text. More than half of the 39 students (20) did not end on a web page displaying a full- 
text scholarly article during the process of search activity, even though they considered 
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themselves to have completed the search successfully. One possible explanation is that 
students do not know what do full-text scholarly articles look like. Additionally, our 
results showed students who use the advanced search features might not find a scholarly 
full-text article. 


Another possible reason that of students did not find a full-text scholarly article is 
that students did not care about the correctness of search results. This could be one of the 
limitations of the current study since, of course, participating in the library instruction 
session and, specifically, taking part in the hands-on search activity is voluntary. Students 
did not receive an incentive (i.e., course credit or grade) for participation or a 
consequence for not participating seriously. Or it may be possible that students are not 
interested in the given topic of search activity, so they picked the keywords cursorily to 
complete the hands-on activity without careful consideration. 


In summary, the above results for undergraduates suggests a lack of knowledge 
and library skills related to searching for academic research. For example, what full text 
scholarly articles are, what it means to conduct searches for academic research, and how 
to use searching resources to find academic sources. 


Although, as noted above, we gained insights that are suggestive for planning of 
instruction, a limitation of this paper is that the screen-recording results analyzed only 
include students’ in-class searching performance before the library instruction session 
began. Future screen recording research should examine and compare the student’s 
scholarly article searching behavior before and after the library instruction session. 


Conclusion 


As librarians we prepare and develop our instruction each semester, and we hear 
positive feedback. However, we hoped to see about making improvements to library 
instruction based on data, instead of guessing what students should or do in fact know. 
Our qualitative assessment of student search behaviors in a writing intensive course in 
psychology gave us data to support our planning and development of library instruction. 
Focusing on what we learned from screen recordings of student searches, our assessment 
of the data indicated that many undergraduates (even in their second year in a major) did 
not have searching routines that involve choosing key searching terms from a research 
topic and then using basic Boolean search strategies effectively with search interfaces 
such as EBSCO’s advanced searching. It also seemed that we should not assume that 
second year students would be inclined to use a discipline’s database for finding research 
articles. We also saw that many students did not know what a full text research article is. 
We will be sharing our findings with colleagues as we collaborate with them to develop 
instruction that incorporates relevant and effective teaching strategies. 
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Abstract 


Videos are one of the most common media formats for online learning. The impacts of different 
presentation styles on student performance and experiences have not been widely studied before. 
Therefore, this study explores how two commonly used video types, screen-capture with 
narratives vs. cartoon animation, can affect student performance and experiences. A user study 
with first-year college students (n=56) was conducted to examine the effects of both types of 
videos on learning performance and experiences. The study results suggested that both the screen 
capture videos and the conversation-based animation videos were equally effective for the 
immediate learning outcome. However, there was a qualitative difference in the student 
responses. More students who watched the cartoon animation videos included more specific 
library terminology or lingo in their responses. 


Keywords: action research; story-based learning; user study, library instruction; tutorial video 


Background 


The front-line educators, including academic librarians, constantly encounter the 
challenges of selecting proper instructional techniques or media in the various learning 
environments; especially in an asynchronized online environment. Some methods have been 
traditionally applied to compose learning materials, including text, graphic, video, and audio 
(DeVaney, 2009; Lusk & Atkinson, 2007; Mayer, 1999, 2001, 2002; Ploetzner & Lowe, 2004). 
In broad definition, technology for video alone includes films, animations (Rieber, 1990; 
Tversky & Morrison, 2002), and screen-capture instructional technology (Smith & Smith, 2012; 
Walker, 2010; Yuen, 2004). The development of multimedia technology enables educators to 
respond to the challenges and the needs that arise from the evolving circumstances. Incorporating 
animated pedagogical agents is a relatively newer instructional delivery method that facilitates 
learners to obtain information and skills using lifelike characters within a multimedia-learning 
environment (Lusk & Atkinson, 2007). All these pedagogical technology techniques are 
considered as a means to integrate different modalities together to aid learning (DeVaney, 2009; 
Lusk & Atkinson, 2007; Mayer, 2001). 


Attending college is one of the most critical life transitions for a lot of people. In the first 
year of college, students encounter various changes and challenges during their transitions and 
adjustments to a new life. Research suggests that first-year undergraduates had higher levels of 
ongoing and chronic stress when compared to other academic years (Misra & McKean, 2000; 
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Towbes & Cohen, 1996). Additionally, Trautwein and Bosse (2017) pointed out the difficulties 
related to: 1) personal aspect, such as balancing studying, part-time work, and family time, 2) 
studying aspect, such as high-curricular demands and pace, 3) organizational aspect, such as 
adjusting to new teaching and learning environments, and 4) social aspect, such as building new 
peer relationships. With further respect to a student’s study capacity, study results showed that 
first-year students tend to have limited information on literacy or research skills (Gross & 
Latham, 2012; Montgomery & Wray, 2020; Zimu, 2020). Gross and Latham (2012) further 
indicated a deviation between the information on literacy skills of first-year students and their 
self-perceptions of their abilities, which means that they have an exaggerated understanding of 
their own abilities (i.e., they "think" that they possess sufficient information literacy skills, but in 
fact they do not). 


Without proper support to manage the college transition, these challenges and difficulties 
may further impact students’ health and may result in other mental health concerns and 
behavioral issues. These issues included sleep disturbance, binge drinking (American College 
Health Association, 2014; Jensen, 2003; Mallett et al., 2013), and ultimately, affected their 
academic performance and well-being (Singleton & Wolfson, 2009). 


The university library is a helpful resource for students to develop study skills, which 
supports their academic performance. For example, researchers indicated that first-year students’ 
use of library services were positively associated with academic outcomes, engagement, and 
skills (Soria, Fransen, & Nackerud, 2013, 2017). Similarly, Hughes, Hall, and Pozzi (2017) 
reported that the use of library services supported the students’ information needs for academic 
purposes. The above suggests the important role of the university librarian who could play a part 
in the lives of the first-year students. They may stimulate the students’ learning interests, respond 
to their learning needs, and ultimately, support their academic success. 


In response to the rapidly changing learning environments, new instructional strategies 
like story-based animations were employed to redesign a set of library video tutorials. 
Storytelling is an effective form of communication and it has been used for learning for decades, 
if not centuries. In contrast to the conventional screen recorded video tutorials with narratives, 
story-based animation videos convey content information through a conversation style between 
two cartoon characters. In particular, the same content was carried out with an animated story. 
For example, instead of using a librarian’s voice telling how to search a database with screen 
recording, the video was set up as a conversation between two students. The scenario is one 
student would tell another student who was having trouble finding information how to search an 
article from the library’s website. These kinds of situations are real-life stories. 


In this paper, we describe the pilot project, our assessment study on the students’ 
reactions to both types of video presentations and opportunities for future development. 
Specifically, the study explored the following questions: 


(1) What is the impact of a story-based strategy on the user experience when using online 
video tutorials? 
(2) What are the learning outcomes from such a story-based approach? 
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Research Design 


According to Merriam (2002), basic interpretive study can be used to understand the way 
people interpret and attribute their experiences and construct their lives. Therefore, the approach 
was deemed appropriate as the author attempted to learn more about students’ experiences of 
watching library video tutorials in two different styles (screen recording with a single voice 
narrative vs. animation with two voices/conversations). Particularly, this approach allowed the 
study to gain a better understanding of the potential influence of presentation styles on learning 
outcomes. Specifically, the content information they received from the two videos and how they 
received the information was interpreted. 


The Videos 


Two library video tutorials with different visual/presentation styles were displayed to 56 
college freshmen in two separate times. One video was a “conventional” show-and-tell screen 
capture video narrated by a librarian. The other video was conversation-based animation video 
with cartoon characters. The content of both videos were the same, which involved: (1) finding a 
specific database (in this case, PsycINFO) from the library homepage, (2) conducting a keyword 
search for journal articles on given topics, and (3) refining a search to peer review journal 
articles. The lengths of time for the two videos were approximately the same (2 minutes) with 
only a few seconds difference. 


Participants 


The population of the current study is the first-year undergraduate students, who are 
normally the target audience of library instruction. After obtaining IRB approval, an email 
invitation was sent to all the first-year students at a public research university in the Midwest 
region. Interested participants contacted the researcher to schedule a session in a lab to watch the 
videos. Fifty-six students were recruited and were randomly assigned into two groups: one was a 
story-based group. Participants in this group watched the animated video with two characters in 
conversation style, while the other was a non-story-based group. Participants in this group 
watched the screen recording video with single voice narrative. 


Procedure 


In each individual session, the participants were greeted and given a consent form with 
the study information that was being presented. The study information was explained both orally 
and in print (the study information sheet). Participants were allowed time to read through it and 
ask questions if they had any. The research was voluntary-based. The participants could have left 
at any time if they did not wish to participate. For the study, the participants were asked to view 
the video, perform tasks with a worksheet, and then fill out a survey. The primary task was to 
conduct a database search with the given information, which was covered in the tutorial, and to 
then write down the bibliographic information of their choice, specifically the title and author(s). 
In addition, students were asked to recall what the video was about. 
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Data Analysis 


The primary data include students’ performance on the tasks and their responses to one 
open-ended question: what was the video about? Both quantitative and qualitative analysis 
methods were applied in the study. Descriptive analytics and coding were applied to provide the 
insight and the patterns of the students’ performance. 


Interpretation was the general approach to analyze data. Interpretation means producing 
reasonable meaning, which is created by researchers with their best efforts from the social 
phenomena they studied (Hatch, 2002). The goal was to explore the experiences of watching 
different presentation style videos among first-year students. Interpretation helps to understand 
what information the students perceived from the two library video tutorials and how they 
applied the perceived information. 


Descriptive statistics (i.e., mean, frequency, and percentage) and open coding were the 
primary methods for data analysis, which provide insight and discovery to the patterns of the 
students’ performance. Opening coding is a systematic process of sorting the collected data 
through identifying concepts, naming, and categorizing them based on its dimensions in order to 
establish major themes (Corbin & Strauss, 2010; McCaslin, n.d.; Merriam, 2002). Themes and 
sub-themes were produced from the analogous responses of worksheets and recurring behavior 
patterns of the article-search task. The responses recorded in the worksheets were coded from the 
following aspects: the level of detail and the completion of the article-search task. The 
observation notes provided the information on the participants’ performance. 


Coding results were present in the following results section, consisting of students’ 
article-search task performance and their responses of an open-ended question. Peer debriefing 
was used in the study to ensure the validity and credibility of the study and to make sure the 
themes and research questions were aligned. 


Results & Discussions 


A total of 56 students participated in the current study, thirty of them watched a screen 
capture with single voice narrative video, and the others watched a conversation-based animation 
video. Results showed that there was no significant difference in immediate learning outcomes 
between the two types of videos, generally. Coded data from the observation notes and 
worksheets showed that all participants recorded bibliographic information of their article-search 
results. It suggested that they performed a successful database search after watching the tutorials. 
Furthermore, coded data from the worksheets showed that all the students were able to describe 
what the videos were about. 


The level of detailed responses on the worksheets was coded from two aspects, including 
the number of word counts and the degree of key points covered. Results showed that the 
average word counts of the participants’ written responses were 17.7 words. Participants who 
watched the screen capture video used an average of 18.1 words in their answers, and those who 
watched a conversation-based animation video used an average of 17.2 words. The results 
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suggest that there were no significant differences in the number of word counts the students 
wrote regardless of the types of videos the students watched. 


Table | 
Average Word Count of the Open-Ended Question (n=56) 


Watched Video Frequency Average word 
counts 
Screen capture Video 30 18.13 
Conversation-based animation video 26 17.23 


However, some qualitative differences in the degree of key points covered in the 
students’ responses emerged from the coded data. The level of details of key points covered in 
the students’ responses were sorted into five categories/levels. The first category is regarded as a 
very general response, which included the least amount of information related to the video 
watched. Responses that fell into this category meant that no key points (i.e., specific purpose, 
tool, databases, library jargon) were included. A sample response was: “Jt was a tutorial video on 
how to complete a certain task.” Responses that fell into the second category included only one 
key point mentioned in the video. A sample response of the second category was: “The video I 
just watched was about how to navigate and use PsychINFO.” Responses of the third category 
included two key points covered in the videos. A sample response of this category was: “The 
video showed how to research articles on the university library website.” 


Responses that fell into the fourth and fifth categories included more detailed video 
information. Responses of the fourth category consisted of three key points relevant to the 
videos. A sample response of the fourth category was: “The video showed how to get to the 
PsyclInfo site and how to narrow your search.” Similarly, responses in the fifth category 
consisted of three key points, but further included library jargons (1.e., keywords, quotation 
point). A sample response of the fifth category was: “The video was an informative video, that 
showed us how to utilize keywords and databases in order to find the resources to write a 


paper.” 


As shown in Table 2, results suggested that more than half of the worksheet responses 
included video information with a medium degree or less. Twenty of the fifty-six students’ 
responses (37%) fell into the third category, eleven of the fifty-six worksheet responses (20.4%) 
accounted for the second category, and one student (1.9%) response accounted for the first 
category. Additionally, results showed that twenty of the fifty-six responses (37%) accounted for 
the fourth category, and four of the fifty-six responses (7.4%) accounted for the fifth category. 
The results suggested that two in five students could provide the detailed video information, and 
a small portion of the students further used library jargons in their responses. Additionally, 
twenty of the fifty-six worksheet responses (44.4%) covered more detailed information from the 
videos, and four of them further used library jargons in their responses. 
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Table 2 


Level of Key Points Covered in Students’ responses (n=56) 


Groups/types of videos watched Frequency % 

Category 1 1 1.9 
Screen capture Video 
Conversation-based animation video 1 

Category 2 11 20.4 
Screen capture Video 9 
Conversation-based animation video Z 

Category 3 20 37.0 
Screen capture Video 2 
Conversation-based animation video 11 

Category 4 20 37.0 
Screen capture Video 11 
Conversation-based animation video 9 

Category 5 4 7.4 
Screen capture Video 1 
Conversation-based animation video 3 


An interesting theme emerged when further looking into the level of details from their 
recall about the video content, and the types of video watched. Results showed that three of the 
four responses in the fifth category (1.e., responses that covered the most video details and library 
jargon) were from students who watched the conversation-based animation video, and one was 
from those who watched the screen capture video. On the other hand, nine of the eleven 
responses of the first category came from students who watched the screen capture video, while 
two came from those who watched the conversation-based animation video. It is uncertain if the 
students were more impressed with the conversation-based animation video or if they had 
already known the video content very well. Further research is needed to explore the association 
between the presentation types of videos and students’ responses and to clarify the students’ 
background knowledge in research skills and library resources. Furthermore, the one response of 
the first category came from a student who watched the conversation-based animation video. The 
result implies that the student may not have paid attention to the video or did not understand the 
video content. 


The author acknowledges the limitations of the pilot study. For example, the level of 
difficulty on the article-search task may be too easy for the students. Its simplicity may disguise 
the possible impacts of the different presentation style videos on the students’ learning 
experiences. 
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Conclusion 


The pilot study explored the library video viewing experience with actual users, first-year 
students who have relatively few or no university library experience or research skills. In 
summary, the results show that there were no significant differences on the immediate learning 
outcomes between the two types of videos (1.e., screen capture with single voice narrative vs. 
conversation-based cartoon animation). All participants performed a successful database search 
after watching the tutorials. Most of the students were able to describe what the videos were 
about and how they would teach their peers on how to conduct a library database search. Coding 
results of the first open-ended question showed that the majority of the students were able to 
describe what the video was about. 


In conclusion, the study results suggest that both the screen capture video and the 
conversation-based animation video were equally effective. However, there was a qualitative 
difference in the students’ responses. More students who watched the cartoon animation video 
included more specific library terminology or lingo in their responses. 


The unique aspect of this study was to investigate the questions with multiple 
perspectives and with actual first-year students who have relatively few or no university library 
experience. We believe that the conversation-based cartoon animation approach for video 
instruction has great potential to enhance student engagement as well as learning in online and 
blended settings. This study explored the approach in an academic library setting with first-year 
college students. The results of the study can potentially lead to improvements in information 
literacy programs and to improve library practices, especially those pertaining to designing, 
developing, and implementing better, more high-impact and innovative story-based approaches. 
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Teaching with XR (Extended Reality) in Higher Education: 
An Analysis of Student perceptions 


Ahmet Ustun 
& 
Siba El Dallal 
Abstract 


With the increasing rate of shifting the educational methods to the online format, immersive 
technologies such as XR (Extended Reality) can help create a unique active learning experience 
in the new online/hybrid format and take it to the next level. This qualitative study explores the 
potentials of XR— particularly the Virtual Reality (VR) technology in higher education to enrich 
student’s experiences. The perspectives of students at a public university were collected during the 
Spring 2019-2020 semester. The findings suggest that VR technology provides positive 
opportunities to enhance student learning experiences, including, but not limited to, promoting 
creative thinking, offering effective learning, encouraging learning by doing, creating motivating, 
entertaining, and intriguing learning environments, in spite of its high costs, potentials physical 
hazards and accessibility challenges. Recommendations for effective implementation were 
identified. Directions for future research on XR in online and hybrid learning were also discussed. 


Keywords 
XR (Extended Reality), Virtual Reality (VR), Immersive Technology, Emerging Technology, 
Higher Education, Technology Innovation. 


Introduction 


The use of immersive extended reality (XR) is proven to be an effective way to augment 
traditional forms of pedagogy (Brown et al., 2020). XR education evolves widely in higher 
education in the past few years as a field that provides unique hands-on experiential learning and 
helps motivate students to achieve the desired skills and knowledge. Research studies revealed a 
link between the usage of XR technologies and improvements in students' academic performance 
and motivation, students’ social and collaborative skills, in addition to students' psychomotor and 
cognitive skills. With the transition to an online format, XR technologies can help bridge the gap 
between the in-person and online courses and create a space for more collaboration and presence 
opportunities. However, empirical research on its applications in higher education is still 
inadequate (Radianti, Majchrzak, Fromm & Wohlgenannt, 2020). 


With the recent development, the ease of use, the proven dramatic impact on pedagogy 
and the noticeably decreased cost of the XR equipment, implementing XR technologies as 
learning tools becomes one of the most active fields of innovation and research among the 
various higher education institutions (Pomerantz, 2019). XR technologies offer learners a unique 
learning experience that may be unattainable in a traditional educational format. This paper 
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reveals the perspectives of students, who were pre-service teachers at the Faculty of Education at 
a public university, of their VR experience in the several courses they studied during the spring 
semester of the 2019-2020 academic year to develop a deep understanding of the usage of VR in 
higher education within its various modalities. The findings discuss the effectiveness of VR 
technologies in creating unique learning experiences to meet the learning goals and how these 
technologies can help achieve them to enhance student learning and engagement. 


Methodology 
Research Design 


A qualitative research method was employed to examine the opinions of pre-service 
teachers of utilizing virtual reality (VR) applications for educational purposes at a public 
university. The aim of using a qualitative research method was to obtain a deep understanding of 
the participants’ perspectives about using virtual reality in educational environments. Yildirim and 
Simsek (2008) indicate that this method allows researchers to gain in-depth information about the 
subject holistically. 


Participants 


The research participants were 17 university students who took an elective course during 
the spring semester of the 2019-2020 academic year. Participants were sophomores at the 
Faculty of Education in a public university, studying different academic programs, including 
Mathematics, Turkish Language, Counseling and Guidance, and Psychological Counseling. 


Data Collection 


A semi-structured interview form was developed and employed by researchers to 
determine the opinions of preservice participants in utilizing VR applications in higher education. 
Experts in the learning Design and Technology field assessed the interview questions, and based 
on their revision and suggestions, the interview questions were modified and finalized. 


Data Analysis 


Thematic analysis method (Hsieh & Shannon, 2005) was used to analyze the collected 
data. Data from in-depth semi-structured interviews was analyzed after transcription through 
manual coding using the inductive approach. 


In order to interpret the participants’ perspectives in an understandable manner, specific 
themes were developed by grouping the codes and then interrelating them to establish categories. 
Meaningful integrations are made in the analysis to describe the findings (Yildirim ve Simsek, 
2013). For the coding process, thematic coding was used to identify the main themes emerging 
across the 17 interviews. Codes were grouped under themes and sub-themes and interrelated to 
develop the findings categories. The validity of the qualitative results was constructed by 
applying three validation strategies (Creswell and Clark, 2017), including member checking, 
triangulation of the data from several resources, and asking other researchers to examine the data 
and review the results. To check the coding reliability, the formula (number of codes agreed by 
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researchers / total number of codes) by Miles and Huberman (1994) was used. The reliability 
percentage of the coding between coders was determined as %89. 


Findings 


Pre-service teachers’ opinions about the use of virtual reality were analyzed through 
thematic coding in accordance with the aim of the research. The themes and sub-themes that 
emerged as a result of examining their opinions of using virtual reality applications for 
educational purposes were identified. The findings from this study were organized into four 
themes: usability, benefits, uniqueness, and drawbacks, as presented in Table 1 


Table 1. Students' Views of Virtual Reality Experiences 


Themes Sub-Themes 


Number 


of 
Students 


Examples of Participants’ Opinion 


Usability Easiness 


Difficulty 


1 


P-6 “Experiencing VR is easy to use and 
entertaining. I did not encounter any negative 
aspects of using VR”. 

P-10 “Those who will use VR applications should 
have knowledge about the use of technological 
equipment”. 

P-11 “The downside is that VR is difficult to use; 
sophisticated equipment is needed to use it”. 


Benefits Creative 
Thinking 


Effective 
Learning 


Abstract to 
Concrete 


Motivation 


P-8 “The use of VR can inspire creative learning”. 
P-11 “VR inspires creative learning. It provides 
opportunities for the development of new 
understandings and perspectives... It offers 
different perspectives and creativity”. 

P-1 “VR increases the retention of learned 
information”. 

P-3 “VR offers better learning opportunities for 
students by creating visual memories”. 

P-5 “A student cannot fully visualize the shape of 
geometric objects in his mind so he can often give 
incorrect answers in response to questions related 
to 3D objects. However, this problem can be 
solved with VR applications”. 

P-16 “VR can make the abstract concepts tangible, 
which cannot be done with other technological 
tools”. 

P-12 “VR can play a more instructive, attractive 
and encouraging role for students in their 
learning”. 

P-16 “I can use VR to motivate students who are 
preparing for the national exam”. 
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Active 4 P-9 “VR encourages passive students to actively 
Learning participate”. 

P-13 “VR provides experiences of learning by 

doing, which promotes the retention of learned 

knowledge”. 

Reinforcement 1 P-9 “VR can be used in areas where students have 

a lot of misconceptions about a subject to correct 

them or it can be used in lessons in order to 

reinforce the topics they learned”. 

Entertainment 7 P-2 “More enjoyable learning can be achieved with 

3D animations”. 

P-11 “Interesting and entertaining games can be 

created with VR for solving mathematical 

problems such as four operations”. 

Intriguing 7 P-10 “VR can be used to attract attention in the 

introduction part of the lesson and provide detailed 

information in the development part”. 

P-14 “I can use VR because its applications are 

attractive”. 

Uniqueness 14 P-4 “By implementing VR, I can take a student to a 

more comfortable environment which can be 

supportive of the treatment process instead of 

taking the student to a usual guidance service 

room”. 

P-7 “It is possible to have students enter a virtual 

room with geometric objects”. 

P-8 “We can watch famous painters who lived in 

the past while drawing in their painting room as if 
they are still alive”. 

Drawbacks Cost 11 p-13 “Various necessary technologies such as 
processor, sufficient memory and graphics are 
highly expensive for VR”. 

P-15 “There are a lot of advantages, but the VR 
headsets are expensive to purchase, which seems to 
be a disadvantage”. 

Conditions 5 P-2 “VR technology can be affected by 
environmental conditions very quickly, thus 
affecting its efficient operation”. 

P-13 “There are certain conditions such as having 
suitable light and quiet environments to 
experience”. 

Accessibility 6 p-9 “VR devices cannot be accessible to 
everyone”. 

P-12 “In the current economic situation, VR usage 
causes the inequality of opportunity because of the 
difficult accessibility of VR”. 
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Physical hazard 4 P-8 “VR may have harmful effects on our eyes 
such as visual impairment, strabismus and so on”. 
P-11 “You may drop and hurt yourself while 
experiencing VR on your own”. 


Table 1 showed 14 sub-themes under the four main themes— usability, benefits, 
uniqueness, and drawbacks. 


Usability This theme refers to pre-service teachers’ thoughts if they were able to use VR 
applications easily. Although one student emphasized the ease of using VR applications, 20% of 
the students (n=4) indicated that integrating VR in education was difficult and required lots of 
technical support and knowledge. 


Benefits: This theme refers to the advantages that VR offers when it is utilized for 
educational purposes, and it's classified into eight sub-themes consisting of creative thinking 
(23%), effective learning (23%), abstract to concrete (47%), motivation (35%), active learning 
(23%), reinforcement (6%), entertainment (41%), intriguing (41%). Creative thinking refers to 
students' ability that is promoted by the VR technologies to find a solution in an original way. 
23% (n=4) of the participants identify creative thinking as one of the main benefits of adopting 
this technology in education. VR presents students with personalized, hands-on learning 
activities and plays an important role in helping students to think critically. The collaboration 
and immediate feedback can help students to improve their critical thinking and their learning 
outcomes. 23% (n=4) of the participants also believed that VR has the potential to 
encourage effective learning, which refers to accommodation and assimilation of new 
knowledge and ideas permanently and concretely. Moreover, about 50% (n=8) of the participants 
believed that VR helps students enhance a tangible understanding of abstract concepts, which 
we refer to by Abstract to concrete. The terms motivation refers to the use of VR to motivate 
students to learn enthusiastically; 36% (n=6) of the interviewees identified this benefit. Active 
learning refers to student engagement in the learning process in an active manner by VR; 23% 
of participants found active learning important, whereas only 6% of the participants (n=1) only 
identified the reinforcement. Reinforcement refers to the VR effects to strengthen student 
learning or to make their misconceptions corrected. Both Entertainment & Intriguing were 
identified as the most important benefits of VR. Entertainment refers to the VR effects in making 
the learning fun. Whereas, Intriguing refers to the fact that VR induces students to know more. 
80% (14 out of 17) of the participants claimed that VR offers a unique learning experience. 


Uniqueness: It refers to the unordinary learning opportunities that VR provides students 
with. VR can offer an authentic, exhilarating, and lifelike learning experience. 


Drawbacks: This theme refers to the disadvantages that students might encounter when 
utilizing VR in the educational process. Four sub-themes emerged under this category, including 
the high cost of VR technologies, accessibility, physical hazards, in addition to the 
environmental context. 


More than 60% of the participants found the high cost of the VR technologies, including 


the cost of its headsets, powerful computer peripherals, and the VR applications, was one of the 
biggest challenges that encounter the higher education institutions. The cost refers to the amount 
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students have to pay to obtain a VR headset and buy its software. 30% of participants identified 
providing the environmental conditions is important to create an effective learning experience. 
Conditions refer to the inside or outside environmental context. VR can be utilized best when 
certain inside or outside environmental conditions occur. 35 % of the participants identified 
accessibility as a challenge for implementing this technology in education. Accessibility refers to 
VR access challenges in which access to this technology cannot widely be acquired because of 
its availability and cost. And a little more than 20% of participants found the Physical hazards 
resulted from the implementation of this technology help prevent this technology spread. 
Physical hazards refer to the possibility that students might accidentally hurt themselves because 
they cannot be aware of their surroundings while using VR technologies. 


Discussion 


This research was carried out to reveal the opinions of students who were pre-service 
teachers about utilizing VR applications for educational purposes at a public university. Based on 
the data analysis, VR has its unique benefits as well as drawbacks when used in educational 
settings. According to their perspectives, the use of VR applications provides opportunities for 
students to learn by doing. It creates an effective virtual learning environment where students are 
engaged in learning activities that are interesting, entertaining, and informative. This kind of 
learning environment motivates students to become active knowledge seekers. Besides, students 
easily comprehend abstract concepts in these virtual learning environments where abstract 
concepts become concrete objects. It can be stated that the use of VR offers instructive, 
attractive, and encouraging learning experiences for students. In parallel with these findings, 
Hussein, and Natterdal (2015) identified the benefits of VR in geology, medical studies, history, 
engineering, and so on. They showed that VR offers interactive learning environments in which 
students are active in their learning, such as performing an experiment in a virtual science lab or 
exploring the ancient world by traveling to places. Also, they indicated that students are 
encouraged to be creative in virtual learning environments where they can find opportunities to 
demonstrate their creative abilities in architecture and other fields. Furthermore, Kaminska et al. 
(2019) identified several advantages of using VR in educational settings, including increasing 
student engagement, providing interactive and attractive learning opportunities, maximizing the 
effectiveness of the learning environment, and encouraging students to be active in their 
learning. A study conducted by Makransky and Lilleholt (2018) revealed that the use of VR 
makes the learning environment enjoyable, engaging, and motivating with great features of VR. 
Ustun, Yilmaz, and Karaoglan Yilmaz (2020) found that students' performance and motivation 
can be maximized by utilizing VR applications. Besides, pre-service students also indicated that 
VR offers unique learning experiences. Providing unique learning experiences can be considered 
as a significant advantage of VR (Kaminska et al., 2019). Pre-service students emphasized the 
ability of VR as an educational tool that provides a 3D virtual learning environment. It cannot be 
possible to create such a virtual environment and promote learning without VR technology. They 
indicated that students could go to historical places regardless of their locations, meet famous 
scientists, artists, and any person who left a mark in history, and experience anything that we 
can't partake in the real world. 


On the other hand, the study findings identified the disadvantages of VR. The main disadvantage 


was its cost. Pantelidis (2010) highlighted the high costs of VR as a primary drawback. The VR 
high cost is one of the main factors that prevent utilizing VR as an educational tool. The findings 
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also indicated that the accessibility of VR technology might cause inequalities in technology 
usage because many students cannot obtain these technologies. The use of technology in 
education is supposed to promote educational equity (Warschauer & Matuchniak, 2010); 
however, VR usage deepens the educational inequity in the society, especially in underdeveloped 
countries. Moreover, studies revealed that the extensive usage of VR might have harmful effects 
on human health such eye disorders, including visual impairment or strabismus, in addition to the 
potential dizziness and nausea after using the headsets. 


Conclusion 


Extended reality (XR) immersive technologies have massive potentials to enrich teaching 
and learning in Higher education through their abilities to offer a unique hands-on experience 
that might be unattainable previously with other educational technologies. The latest generation 
of XR technologies such as Oculus Quest 2 showed great improvement and enhanced 
performance; they are relatively affordable, wireless, less nauseating, and provide a safer 
experience. 


References 


Brown, M., McCormack, M., Reeves, J., Brooks, C., & Grajek, S. (2020). EDUCAUSE Horizon 
Report. Teaching and Learning Edition. Louisville, CO: EDUCAUSE. 


Hsieh, H.-F., & Shannon, S. E. (2005). Three Approaches to Qualitative Content Analysis. 
Qualitative Health Research, 15(9), 1277-1288. https://doi.org/10.1177/1049732305276687 


Yildirim, A. & Simsek. H. (2013). Sosyal Bilimlerde Nitel Arastirma Y6ntemleri. [Qualitative 
research methods in the social sciences.] Seckin Yayincilik. Ankara. 


Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook 
(2nd ed.). California: SAGE Publications 


Tunur, T., Hauze, S. W., Stuhr, P. T., & Frazee, J. P. (2020). The Impact of XR-Immersive Labs 
on Student Motivation to Learn Kinesiology (No. 3233). EasyChair. 


Pomerantz, J. (2019). Teaching and Learning with Extended Reality Technology. Information 
and technology transforming lives: connection, interaction, innovation. 


Martin-Gutiérrez, J., Mora, C. E., Afiorbe-Diaz, B., & Gonzalez-Marrero, A. (2017). Virtual 
technologies trends in education. EURASIA Journal of Mathematics, Science and Technology 
Education, 13(2), 469-486. 


Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of 
immersive virtual reality applications for higher education: Design elements, lessons learned, 
and research agenda. Computers & Education, 147, 103778. 


Parsons, T. D., & Trost, Z. (2014). Virtual reality graded exposure therapy as treatment for pain- 


related fear and disability in chronic pain. In Virtual, augmented reality and serious games for 
healthcare | (pp. 523-546). Springer, Berlin, Heidelberg. 


298 


Warschauer, M., & Matuchniak, T. (2010). New technology and digital worlds: Analyzing 
evidence of equity in access, use, and outcomes. Review of research in education, 34(1), 179- 
225, 


Pantelidis, V. S. (2010). Reasons to use virtual reality in education and training courses and a 
model to determine when to use virtual reality. Themes in Science and Technology Education, 
2(1-2), 59-70. 


Hussein, M., & Natterdal, C. (2015). The benefits of virtual reality in education: A comparison 
study. (Bachelor thesis). Chlamers University of Technology, University of Gothenburg, 
Goteborg, Sweden. Retrieved from http://hdl.handle.net/2077/39977 


Kaminska, D., Sapinski, T., Wiak, S., Tikk, T., Haamer, R. E., Avots, E., Helmi A., Ozcinar C., & 
Anbarjafari, G. (2019). Virtual reality and its applications in education: Survey. Information, 
10(10), 318. 

Makransky, G., & Lilleholt, L. (2018). A structural equation modeling investigation of the 
emotional value of immersive virtual reality in education. Educational Technology Research and 
Development, 66(5), 1141-1164. 


Ustun, A.B., Yilmaz, R., Karaoglan Yilmaz, F. G. 2020. Sayfa 56. “Virtual reality in medical 
education”. In Sajid U. (Ed.), Mobile Devices and Smart Gadgets in Medical Sciences, IGI Global. 
DOI:10.4018/978-1-7998-2521-0. 


Authors 


Ahmet Berk Ustun (Ph.D.) is currently an associate Professor at Bartin University, Turkey. He 
earned his PhD in Learning Design and Technology from Wayne State University, Detroit, 
Michigan. He received his master’s degree in Computer Education and Instructional 
Technology from Gazi University, Turkey and his bachelor’s degree in Computer Education 
and Instructional Technology from Selcuk University, Turkey. His research interests are in 
online and blended learning, mobile learning, emerging technologies and MOOCs. 


Email: ustun.ab@gmail.com 


Siba E] Dallal (Ph.D.) is currently a Learning and Teaching Technology consultant at the College 
of Literature, Science and the Arts (LSA)- University of Michigan- Ann Arbor, where she partners 
with LSA faculty utilizing technology to enhance teaching and learning processes. She is a 
Ph.D. graduate in Learning Design and Technology program at Wayne State University. She 
earned two master’s degrees in Learning Design and Technology from Wayne State University 
and Interdisciplinary Studies and leadership from Buffalo State University- SUNY-NY. Prior to 
U-M, she was a graduate research assistant and an adjunct faculty at Wayne State University. She 
is extremely interested in emerging technologies in Education. Her research work is focused on 
technological change management, LMS and the effective implementation of emerging 
technologies in education. 

College of Literature, Science, and the Arts | University of Michigan 

2014 Modern Languages Building | 812 E. Washington Street | Ann Arbor, MI 48109 

Email: sibael@umich.edu 


299 


Design Considerations for Learning, Abilities and Cultures: 
Progress Monitoring Software for Learning English or Sign Language 


Sudip Kumar Ghosh 
Department of Learning and Performance Systems 
Pennsylvania State University 
314 Keller Building, University Park, PA 16802 
Email: skg91@psu.edu 


Simon Richard Hooper 
Department of Learning and Performance Systems 
Pennsylvania State University 
314 Keller Building, University Park, PA 16802 
Email: sxh12@psu.edu 


Jian Liao 
Department of Learning and Performance Systems 
Pennsylvania State University 
314 Keller Building, University Park, PA 16802 
Email: leojames123@gmail.com 


Susan Rose 
Department of Educational Psychology 
University of Minnesota 
EdSci Building, 25 East River Road Minneapolis, MN 55455 
Email: srose@umn.edu 


Rayne Audrey Sperling 
Department of Educational Psychology, Counseling, and Special Education 
Pennsylvania State University 
278 Chambers Building, University Park, PA 16802 
Email: rsd7@psu.edu 


Introduction 


As researchers, we rely on good empirical evidence to design educational technology for 
schools. However, when educational technology is implemented in schools, challenges may arise 
depending on technology considerations, people’s background or expertise, and cultures 
(Blumenfeld et al., 2000). Further changes may be needed after technology is validated with 
theory or empirical evidence to ensure that they are suitable for use in the schools where such 
technology is deployed. This paper discusses some of the changes that were made to a progress 
monitoring software, AvenuePM, to ensure that the software is suitable for teachers to use in 
classroom settings. 


AvenuePM is a web-based progress monitoring tool that was developed with the aim of 
helping teachers to track the literary progress of students who are learning either American Sign 
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Language (ASL) or English. Originally, the software was developed as a digital tool for teachers 
who were monitoring the progress of students who were Deaf or Hard or Hearing (DHH). Over 
more than a decade, the software has gone through many design iterations (Hooper et al., 2013; 
Miller et al., 2008) to make it more useful to schools — teachers, students, parents, and others. 
Today, the software is used in elementary schools across the United States that have diverse 
student populations with different abilities. 


In this paper, we discuss two studies related to the design and development of the 
software. The aim was to gather usability feedback from participants and make design changes 
so that teachers find the software easy to use and easy to learn. The first study was a pilot study 
with undergraduate students as participants. The pilot study was used to verify the research 
protocols and the use of usability metrics, such as System Usability Scale (SUS) (Brooke, 1996) 
and Singe Ease Questionnaire (SEQ) (Sauro, 2012), to evaluate the software. Feedback from the 
study was used to make design changes before the second study. Teachers participated in the 
second study and further evaluated the software. Findings from the studies, along with the design 
changes, are presented in this paper. 


Progress Monitoring and Design 


Progress monitoring is a form of formative assessment where the goal is to improve 
student learning (Pellegrino, 2019). Formative assessments are used with the aim of helping 
students to learn and achieve learning goals. Depending on the learning goals and the instruction 
strategies, assessments are designed so that students can perform assessment tasks, and data from 
the assessment tasks are interpreted against expected progress levels or learning goals (National 
Research Council, 2001). Data from assessment tasks can be useful to both teachers and students 
as feedback to improve learning (Bransford et al., 2000). In this article, the focus is on making 
design changes so that teachers can easily administer assessments and interpret assessment data. 


AvenuePM consists of eight assessments that are designed using principles of 
Curriculum-Based Measures (CBM; Deno, 1985, 2003); CBM assessments are designed to be 
easy to administer, brief (1 to 5 minutes), reliable, and valid. Teachers administer the 
assessments in the classroom and assessment data are used to monitor student progress against 
expected progress or learning goals. Teachers interpret assessment data to assess student 
progress levels and decide if students are making adequate progress to meet learning goals. To 
understand whether teachers can interpret data, aspects of graph comprehension (Friel et al., 
2001) are relevant — identifying data, recognizing patterns, and making decisions. Interpretation 
of such assessment data are also dependent on contextual factors such as the purpose of 
interpretation, background, expertise, task characteristics, area of expertise, culture, etc. (Coburn 
& Turner, 2011; Friel et al., 2001; Pellegrino, 2019). Many of these contextual factors contribute 
to the challenges of ensuring that technology is useful in schools (Blumenfeld et al., 2000). 
Feedback from teachers can help in making suitable design changes for teachers. 


AvenuePM supports many tasks that help teachers with classroom management, 
assessment administration, assessment scoring, and assessment data interpretation. As we make 
changes to the software, the purpose of design changes is to make better software for schools and 
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teachers — to design a better alternative (Simon, 1988). The research team is trying to make 
changes so that teachers find the software easy to use and easy to learn. The studies below 
describe the process of getting feedback from participants and making design changes. 


Study 1: Pilot 


A pilot study was conducted to assess whether study materials and procedures are 
feasible for evaluating the software. Since participants were undergraduate students, and not 
teachers, the objectives of the study were limited to ease of use and ease of learning; assessment 
data interpretation tasks were not included. 


Method 
Participants 


27 undergraduate students (25 female, 2 male) from a large university in North East 
United States completed the pilot study. Participants received an extra course credit in lieu of 
participation. All participants were unfamiliar with CBM assessments and AvenuePM software. 


Materials 


A web-based survey contained brief instructions on the tasks to be completed with no 
details or steps. The survey contained sections about the study, the institutional approvals, 
consent for participants, participant profile information, a simple one-line instructions for the 
tasks to be completed, comments and ratings of ease of use for each task, and ratings for the 
overall system usability. 


A task analysis was conducted to identify important tasks in AvenuePM related to 
progress monitoring: 6 class management tasks, 5 assessment tasks, and some essential tasks for 
using the software. The complete list of tasks is included in Table 2. A seven-point Likert rating, 
Single Ease Questionnaire (SEQ; Sauro, 2012) was used to collect ease of use ratings for each 
task. System Usability Scale (SUS; Brooke, 1996) was used to collect ratings for the overall 
system usability; SUS is a 10 item questionnaire that is reported on a 0-100 scale. Data on both 
SEQ and SUS allow for comparisons between sample data and other larger data sets (Bangor et 
al., 2008; Sauro & Lewis, 2016). The recommendation is that sample ratings should have a mean 
score that is higher than the median scores reported for SUS or SEQ; the median scores for SUS 
and SEQ is 68 and 5.5 respectively (Bangor et al., 2008; Sauro, 2018). Lewis & Sauro (2018) 
suggest that the desired SUS scores should be 80 or more, which corresponds to a score in the 
top 15 percentile. 


Procedure 


All participants completed the study in the presence of a moderator. The moderator used 
a scripted protocol to maintain consistency across the study sessions. Participants were informed 
of their rights, including their ability to quit at any time, and consent was recorded for 
participation and video recordings. A laptop with internet connection was provided for 
completing the tasks. The moderator took notes during the study and the session was recorded 
with the consent of participants. The study was designed for 1 hour; all participants completed 
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the study in 45 minutes or less. The study time includes the initial briefing session, or other 
related activities. 


Results 


All participants completed each of the assigned tasks within 5 minutes. The mean SUS 
scores for the overall AvenuePM system is 74 (Table 1), which is more than the recommended 
median score of 68 and falls in the 3 quartile of SUS scores (Bangor et al., 2008). 


SEQ ratings for ease of use corresponding to individual tasks are included in Table 2. 
SEQ ratings suggest that common tasks such as logging in, logging out, and finding items were 
rated as easy. However, tasks such as ‘Creating a Teacher Account’ and assessment-related tasks 
such as PictureNaming and Slash were rated as difficult. 


Design Changes 


Results suggested that the AvenuePM system meets the minimum recommended SUS 
score but not the desired score of 80, or more. Based on feedback, labels and instructions for 
assessments were added or simplified. Some participants had asked for practice sections even 
though practice sections were present; labels for practice sections were changed to make them 
clear and easy to find. Short demonstration videos of the assessments were added as help 
materials. However, the label for creating a new account, ‘Request an Account’ (see Figure 1) 
was not changed though the label ‘Create an Account’ is the more widely used. The research and 
design team thought that teacher feedback in a later study will be useful to confirm such details 
for ease of use, and to compare feedback from undergraduate students and teachers. 


Study 2: Remote Study 


The follow-up study was a remote study conducted after the design changes described 
above. Teachers participated in the second study, so data interpretation tasks were added as they 
are important for progress monitoring (Coburn & Turner, 2011; Friel et al., 2001; Fuchs, 2016). 
A conceptual data chart concept was also used in the study to gather feedback from teachers. 
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Figure 1. Initial design of the AvenuePM homepage. 
Method 
Participants 


Participants were 9 teachers (7 female, 2 male) from four different states in the United 
States. All teachers had some knowledge of CBM. Four teachers were from schools that partner 
with the AvenuePM team, and were familiar with the software; the 5 other teachers were using 
the software for the first time. All participants received a 50 dollar gift voucher as compensation. 


Materials 


The materials were similar to those used in the previous study. Two additional tasks 
related to data interpretation (items 6a, 9a in Table 4) were added. In data interpretation tasks, 
the moderator asked the participants to identify the data elements, the trend, and compare the 
progress levels against expected progress levels or goals. 


Procedure 


The study was a remote study conducted using the Zoom video conferencing software. 
The moderator and the participants used their own device and internet connection. All other 
procedures were similar to the previous study. The study was designed for 1 hour and most 
studies lasted between 45 and 60 minutes. 


Results 


Participants successfully completed all tasks, except one; one participant did not 
complete the task of creating a new account as web forms failed to load correctly. A total of 143 
of the attempted 144 tasks were completed successfully. The task completion rate was 99.3%. 
The mean SUS score for AvenuePM was 82 (Table 3), representing a score in the top quartile of 
SUS scores (Bangor et al., 2008). SEQ ratings on ease of use are included in Table 4. Tasks such 
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as logging in, logging out and finding items were rated as easy to use. All tasks had a mean SEQ 
rating of at least 5.5, except the following: creating an account; sharing a student; using data 
charts; and the data chart option. 
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Figure 2. Redesigned home page of AvenuePM. 
Discussion 


Results from the second study suggest that the design changes were effective in 
improving the SUS usability ratings. The SUS mean score of 82.2 corresponds to a grade A, or 
top 10 percentile SUS scores, according to Lewis and Sauro (2018). Though SUS scores may 
improve with further modifications, it may require larger sample sizes to confirm improvements. 
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Also note that though overall usability scores seem good, tasks related to creating an account, 
sharing a student, and date interpretation were not rated as easy to use; there is scope for making 
improvements. While SUS scores assess overall usability, SEQ ratings and feedback on induvial 
tasks can be combined to complement SUS scores and make specific changes (Sauro & Lewis, 
2012). 


Most of the participant feedback was about making labels or instructions clear using 
common language and ensuring that participants can find them. Participants noted that the task 
of creating an account was confusing because of the label, ‘Request an Account;’ most 
participants (undergraduates and teachers) preferred the standard label of “Create an Account’ or 
‘Create a Free Account.’ Some participants noted that the purpose of AvenuePM software was 
not clear based on content on the homepage. For the task of sharing a student, participants noted 
that the labels were ambiguous as the terms ‘share a student’ and ‘change teacher’ were used 
interchangeably in different places; participants were not clear whether sharing a student meant 
sharing student details with another teacher, or changing the teacher for a student, or both. 
Participants preferred consistent design elements like labels, images, or titles across pages. 


In tasks related to assessments and data interpretation, participants appreciated the 
practice sections, help videos and other instructions provided in AvenuePM. However, 
participants again asked for simpler instructions and consistency as they were important tasks. 
For example, if the link or button labeled ‘PictureNaming’ was used to navigate to the 
PictureNaming assessment page, participants appreciated it when the subsequent page title was 
also consistently named PictureNaming and contained simple instructions about the assessment; 
participants preferred having the same label and icon at the top of the next page so that it was 
immediately clear that the new page was the intended destination. Teachers appreciated the 
student assessment tasks. PictureNaming task was rated very easy to use on the SEQ ratings (M 
=6.1, SD = 1.1). Other assessment tasks, Slash and Word Sign/ Say, were also rated above the 
recommended SEQ rating, but participants commented that more instructions were needed. All 
participants successfully completed assessments tasks in the allotted 5 minutes without any 
additional instructions. However, since these are assessment tasks or tests, participants wanted 
clear instructions for the assessment tasks and scoring to prevent any possible ambiguity for 
students. Similarly, though participants interpreted the data charts successfully, some participants 
noted that labels and interfaces could be simpler and more intuitive. The conceptual data chart 
shown was also interpreted correctly by all participants, but existing data charts were preferred. 
Figure 3 shows the redesigned home page after the second study. 


Conclusion 


Results suggest that usability and ease of use score improved in the second study. The 
pilot study suggested that the overall usability of AvenuePM was adequate. Though participants 
of the pilot study were undergraduate students, and not teachers, feedback from undergraduates 
was useful in making design changes for teachers. The SUS usability scores improved from 74 to 
82, corresponding to 70 percentile (grade B) and 90 percentile (grade A) SUS scores (Lewis and 
Sauro, 2018). The individual feedback on tasks along with the corresponding SEQ scores 
provided specific data on making design changes at the task level. Though participants 
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completed most tasks successfully, feedback from teachers suggests that further work is needed 
to ensure that labels and instructions are simple, clear, and relevant; tasks related to assessments 
and data interpretations may need further research on understanding other contextual factors. 
Additional usability studies will help to make the software better and, hopefully, to ensure that 
teachers can use the software effectively for the main purpose of progress monitoring — helping 
students to learn. 


Table 1. System Usability Score (N=27). Fall 2018 Study. 


95% Confidence Interval 


Lower Upper 
Metric Average SD Limit Limit 
System Usability Score (SUS) 74.6 13.8 69.2 80.1 


Table 2. Single Ease Question scores (7-point Likert scale, N=27). Fall 2018 Study. 


95% Confidence Interval 


Lower Upper 
Task Average SD Limit Limit 
1 Creating a teacher account 32 1.6 2.6 3.9 
2 Logging in as a teacher 6.8 0.5 6.6 7.0 
3 Creating a Class 6.4 1.0 6.0 6.8 
4 Adding a Student to a Class 6.4 0.8 6.1 6.8 
5 Changing / editing Student Details 5.3 ie 4.6 5.9 
6 Finding Student data, charts or scores 6.0 1,5 5.4 6.6 
7 Finding notifications/ to-do list 6.7 1.0 6.3 tel 
8 Sharing a student with another teacher 5.1 Leh 4.4 5.8 
9 Logging out as a teacher 6.7 1.0 6.3 12 
10 Logging in as a student 6.9 0.5 6.7 7.0 
11 Finding Picture Naming & Slash as 6.5 0.9 6.2 6.9 
student 
12 Picture Naming - completing a task cm 2.0 23 3.9 
13 Slash - completing a task 34 1.7 3.0 4.4 
14 As student, finding your progress data 4.8 2A 30 5.6 
15 Word Sign/ Say - completing a task 4.5 1.8 3:8 D2 


Table 3. System Usability Score (N=27). Spring-Summer 2019 Study. 
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95% Confidence Interval 


Lower Upper 
Metric Average SD Limit Limit 
System Usability Score (SUS) 84.1 12.6 33 94.6 


Table 4. Single Ease Question Scores (7-point Likert scale, N=27). 
Spring-Summer 2019 Study. 


95% Confidence 
Interval 
Lower Upper 

Task N_ Average SD Limit Limit 
1 Creating a teacher account 9 4.8 1.9 3.4 6.2 
2 Logging in as a teacher 9 7.0 0.0 7.0 7.0 
3 Creating a Class 9 6.8 0.4 6.4 EA 
4 Adding a Student to a Class 9 6.4 0.5 6.0 6.9 
5 Changing / editing Student Details 9 6.8 0.4 6.4 7.1 
6 Finding Student data, charts or scores 9 6.0 1.5 4.9 ited 
6a Using the Data Chart 9 5.0 1st 30] 6.3 
7 Finding notifications/ to-do list 9 6.8 0.4 6.4 7.1 
8 Sharing a student with another teacher 9 5.0 1.9 3:5 6.5 
9 Logging out as a teacher 9 7.0 0.0 7.0 7.0 
9a Data Chart Option 9 5:2 1.5 4.1 6.4 
10 Logging in as a student 9 7.0 0.0 7.0 7.0 
11 Finding Picture Naming & Slash as 
student 9 7.0 0.0 7.0 7.0 
12 Picture Naming - completing a task 9 6.1 1.1 53 6.9 
13 Slash - completing a task 9 Del 1.6 4.5 6.9 
14 As student, finding your progress data 9 6.2 Lot 5.4 7.1 
15 Word Sign/ Say - completing a task 8 5.9 1.4 4.7 7.0 
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Abstract 


This paper presents a review of distance education literature published in the Turkish 
Online Journal of Distance Education (TOJDE) to describe the status thereof and to identify 
gaps and priority areas in distance education research based on a validated classification of 
research areas. The articles (N=784) published between 2000 and 2015 were reviewed for this 
study. Accordingly, computer-assisted content analysis revealed that “the rise of ICT and e- 
learning (2000-2003)”, “the increasing use of educational technologies in distance education 
(2004-2007)”, “technology supported online/virtual learning environments (2008-2011)”, and 
finally “the rise of the student centered, technology supported education and data-driven 
approaches (2012-2015)” were major themes in articles published in TOJDE. 


Keywords: Turkey, distance education, research trends, systematic review, bibliographic 
analysis 
Introduction 


Taking advantage of previous experience and then using it as guide provides a robust 
walkthrough for those who want to move forward and improve the field. In this regard, 
analyzing, understanding and then synthesizing research trends in Distance Education (DE) 
forge the current state of the art and help it to develop further. Similarly, Lee, Driscoll and 
Nelson (2004) emphasize that "understanding trends and issues in terms of topics and methods 
is pivotal in the advancements of research on distance education" (p. 225). The structure of a 
research discipline forms the foundation for identifying gaps and priority areas (Mishra, 1997, 
p. 281). According to Bozkurt and Akgun-Ozbek, (2015), identifying the research areas is 
important because they act like a compass for online distance education researchers. They 
further highlight that distance education is a dynamic, interdisciplinary field that reacts to the 
changes swiftly; therefore, it is vital to keep the knowledge up to date through research on 
global, local or glocal dimensions. In this regard, analyzing, understanding and then 
synthesizing research trends in Distance Education (DE) forge the current state of the art and 
help it to develop further. In this regard, this paper addresses following research questions with 
a special emphasis on Turkish distance education research: 


e What are the most common research areas covered in TOJDE and where are there gaps 
in distance education research? 
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Literature Review 

Latchem (2009) conducted a content analysis of articles published in TOJDE between 
2000 and 2008. He reported that articles originated from the Asia followed by articles from the 
Middle East, Africa, South America, USA, Eastern and Western Europe, and Australia. He 
highlighted that TOJDE is an important source to better understand the non-Western 
perspectives in distance education field. Ozarslan, Balaban-Sali and Demiray (2012) analyzed 
the articles in TOJDE published between 2000 and 2010. They reported that Turkey, USA, 
India, Nigeria, Malaysia, Pakistan, Australia, Canada, UK, Bangladesh, Greece, and Iran, 
respectively, are the leading contributors. Bozkurt, Zawacki-Richter and Aydin (2019) 
conducted social network analysis to identify keyword network patterns and revealed that 
technolology centric views widely accepted in the articles published in TOJDE. Aydin, 
Zawacki-Richter and Bozkurt (2020) conducted a follow up study and reported that as an 
international open access journal, TOJDE is representative of its own region, mostly developing 
countries, and is also representative of many other countries, which makes it an important 
publication venue. 


In addition to above studies, there are some other papers that examine distance education 
from the perspective of scholarly journal networks. For instance, Zawacki-Richter, Anderson, 
and Tuncay (2010) examined impact of 12 distance education journals (6 open and 6 published 
in closed format). They found that articles in open access journals, such as TOJDE, tend to be 
cited more quickly than in closed format journals. Zawacki-Richter and Anderson (2011) 
analyzed the relationships and influences in peer reviewed distance education journals using 
social network analysis. In their research, they found that TOJDE is one of the journals in the 
core of the citation network. Lastly, Perkins and Lowenthal (2015) investigated open access 
journals in educational technology and reported that TOJDE is one of the most influential open 
access journals. 


There were some other efforts, which were intended to map out trends in distance 
education research and scholarship by examining journals. For instance, Zawacki-Richter and 
Naidu (2016) analyzed titles and abstracts of the 515 articles, published in the SSCI journal 
Distance Education between 1980 and 2014, and they identified seven emerging themes: 
Professionalization and institutional consolidation (1980-1984), instructional design and 
educational technology (1985-1989), quality assurance in distance education (1990-1994), 
student support and early stages of online learning (1995-1999), the emergence of the virtual 
university (2000-2004), collaborative learning and online interaction patterns (2005-2009), 
and interactive learning, MOOCs and OERs (2010-2014). Zawacki-Richter, Alturki and 
Aldraiweesh (2017) examined 580 articles published in International Review of Research in 
Open and Distance/Distributed Learning IRRODL) between 2000 and 2015 and they revealed 
following emerging themes: the establishment of online learning and distance education 
institutions (2000—2005), widening access to education and online learning support (2006— 
2010), and the emergence of Massive Open Online Courses (MOOCs) and Open Educational 
Resources (OER) (2011-2015). 


In addition to these studies, some other articles examined distance education journals 
from a broader perspective and sampled more than one journal in their research. Berge and 
Mrozowski (2001) examined 890 articles published over a ten-year period from 1990 to 1999. 
They reported that key issues in distance education were the roles of key participants, 
technology selection and adoption, design issues, strategies to increase interactivity and active 
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learning, learner characteristics, learner support, operational issues, policy and management 
issues, equity and accessibility, and cost/benefit trade-offs. Lee et al. (2004) also examined 383 
articles published in four distance education journals between 1997 and 2002. They indicated 
that that interaction, learners, perception, collaboration, video conferencing, program 
evaluation, and faculty support were main issues covered in distance education journals. 
Zawacki-Richter, Backer and Vogt, (2009) analyzed 695 articles published in five distance 
education journals between 2000 and 2008. They found that issues about instructional design, 
interaction and communication patterns in computer-mediated communication, learner 
characteristics, and educational technology dominated the distance education field. Bozkurt et 
al., (2015) conducted a complementary study and examined 861 articles published in seven 
distance education journals published between 2009 and 2013. They confirmed that educational 
technology, interaction and communication in learning communities, learner characteristics, 
and instructional design are the most studied research areas. 


Methodology 
Research design 
This study is a literature review that intends to reach a synthesis by examining articles 
published in TOJDE. For this purpose, review study benefits from computer-assisted content 
analysis. 


Computer-assisted content analysis 

Computer-based content analysis enables us to examine the conceptual structure of text- 
based information, so it can be used to identify the most important and most commonly 
occurring themes within large bodies of text (Krippendorf, 2013). This approach is considered 
to be an appropriate method for mapping out a research domain (Fisk, Cherney, Hornsey, & 
Smith, 2012). By employing this approach, Leximancer, the software tool creates concept maps 
that display the core concepts within the text body (conceptual analysis) and show how these 
concepts are related to each other (relational analysis) by recording the frequency with which 
words co-occur in the text. Similar concepts that appear in close proximity are clustered 
together in the concept map (Smith & Humphreys, 2006). In this research, titles and abstracts 
of the articles published in TOJDE analyzed to identify research themes. 


Sample: Articles published in TOJDE 
For this study, all the articles published in TOJDE between 2000 and 2015 were 
reviewed (N=784). Book reviews and editorial notes were excluded from the sample. 


Results and Discussion 
The Rise of ICT and e-learning (2000-2003) 

The major themes emerging between 2000 and 2003 are distance (100%), learning 
(23%), study (18%), information (13%), developing (12%), teachers (8%), and process (5%) 
(see Figure 4). Developments in ICT and online technologies paved the way of e-learning. 
Distance education mostly benefited from e-learning model and it is used for facilitating 
learning processes. Students’ perceptions and teachers’ views on e-learning, and comparison of 
traditional courses and online courses were other topics covered in this period. 


By the 2000s e-learning was becoming widely accepted and adopted by higher education 


institutions (Guri-Rosenblit, 2009) and thought to be as an innovative component of the distance 
education (Bates, 2001). E-learning was further considered as a natural evolution of distance 
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learning (Garrison and Anderson, 2003). As a reflection to these thoughts, many studies 
published in TOJDE between 2000 and 2003 dealt with issues concerning ICT and e-learning 
from the perspective of distance education. This situation can be seen in Figure 4 (see concept 
path communication — information — technologies — distance). As examples of the articles 
published in TOJDE, Reddy and Srivastava (2003) explained ICT and the future of DE; Rajesh 
(2003) researched on problems associated with ICT adaptability in developing countries; 
Edmundson (2003) evaluated ICT in terms of cultural disparity; Flood (2002) suggested that e- 
learning is a driver for continuing professional development; and Fourmier reported the results 
of their survey on e-learning in Europe. In this context, students’ perceptions and teachers’ 
views were also important themes in this term (See concept path teachers-research-offered- 
paper-students-perceptions in Figure 4). For instance, Nakos and Jourdan (2002) explored 
students’ perceptions of online courses, while Dzakiria and Idrus (2003) examined teacher- 
learner interactions in distance education. 
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Figure 4: Concept map for the period between 2000 and 2003 


The theme, rise of the ICT and e-learning, is similar to the themes identified in the 
journals Distance Education and IRRODL. Zawacki-Richter and Naidu (2016), who examined 
the journal of Distance Education, labeled the years between 2000 and 2004 as the emergence 
of the virtual university and Zawacki-Richter et al. (2017), who examined IRRODL, labeled 
the years between 2000 and 2005 as online learning and distance education institutions. Based 
on these findings, it can be concluded that online learning had already moved into the 
mainstream of distance education by the beginning of the new Millenium. 


Increasing use of educational technologies in distance education (2004-2007) 

The major theme between 2004 and 2007 was learning (100%), followed by education 
(83%), technology (34%), program (16%), training (10%), and group (6%). When compared, 
we can see a strong focus on learning which intersects with all other themes except training (see 
Figure 5). As identified in the previous period, as a result of the developments in ICT and the 
great interest in e-learning, educational technologies in distance education attracted much 
attention. As a field of practice, distance education naturally intended to increase learning 
opportunities by benefiting from educational technology. 
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From the beginning of distance education, educational technology and distance 
education has an intertwined relationship (Casey, 2008). Even though educational technology 
is not a new area, it has evolved and took many forms (Reeves, 2006). By 2004, Web 2.0 
emerged and the nature of online learning spaces changed in line with these developments. 
Therefore, as an extension of the previous time period, rise of the ICT and e-learning, articles 
published between 2004 and 2007 revisited educational technology to explore new 
developments from the view of distance education. As can be seen in figure 5, /earning, e- 
learning, technology, information, technology, and knowledge concepts are directly linked to 
each other and demonstrated the direction of research interest in this time period. Some of the 
noteworthy articles published in TOJDE between 2004 and 2007 are as follows: Ozana (2007) 
examined attitudes of graduate students on educational technology and distance education. 
Additionally, it was also seen that among the many educational technologies, there was a focus 
on Learning Management Systems (LMSs) in the articles published in TOJDE between 2004 
and 2007. In their article, Sturgess and Nouwens (2004) evaluated online LMSs from an 
Australian context, Ahmad, Edwards and Tomkinson (2006) examined use of LMSs in distance 
learning from a United Kingdom perspective. 


environment 


work 
p GROUP 
Internet i! 


| sk group 
technology | 7 
informatiqaarning 
based y 
process TECHNOLOGY 
dC < 


ita 
developmen} : A used dal 
using 
time 
LEARNING niversity 
students Wi 
training | study 
access ge, 
TRAINING Saar research, 
differe 


y 
institutions _ education 
educational program 


EDUCATION PROGRAM 


aor, 


Figure 5: Concept map for the time period between 2004 and 2007 


While an increasing use of educational technologies in distance education was 
identified as the main theme for the 2004-2007 period in TOJDE articles, for the Distance 
Education journal it was collaborative learning and online interaction patterns in 2005-2009 
(Zawacki-Richter and Naidu, 2016) and widening access to education and online learning 
support in IRRODL for 2006-2010. 


Technology supported online/virtual learning environments (2008-2011) 

The third theme emerged as an extension of the previous periods. Accordingly, students 
(100%), education (88%), learning (74%), academic paper (37%), information (18%), 
communication (10%) and data (4%) were the themes identified for the articles published 
between 2008 and 2011 (see Figure 6). As can be seen in the concept map, higher education 
institutions used online learning environments to support the learning processes and lessen the 
limitations of distance education, which is mainly about time and space constraints, by using 
online spaces in learning processes. As can be seen in figure 6, learning, distance-learning- 
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online-course-environment concepts are directly linked to each other and demonstrated the 
direction of research interest in this time period. We also see that data emerged as one of the 
major themes, which then became very critical in the following period. The interest 
online/virtual environments stem from one of the popular online/virtual environment, 
SecondLife at this time period and 2011 issues mostly published articles about online/virtual 
environments and worlds. For instance, in their study, Inman, Wright and Hartman (2011) 
evaluated use of Second Life from the perspective of K12 and higher education; Liski (2011) 
examined Second Life for training design, and Hargis evaluated Second Life for distance 
education. 
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Figure 6: Concept map for the time period between 2008-2011 


While online learning environments were not a new phenomenon, they became a viable 
learning space with many new concepts, such as semantic web, social networking, and cloud 
computing. Unlike other developments in technology, online learning in networked spaces 
devised its own pedagogy, which was then called the third generation of distance-education 
pedagogy (Anderson and Dron, 2011) and more specially known as connectivism (Siemens, 
2004). With the opportunities that online learning environments afford, separation in time and 
space is no longer an issue; in contrast, the new motto with online learning is anytime, anywhere 
learning (Ally, 2004). Based on these developments in the field, not surprisingly, one of the 
major issues covered in the studies published in in TOJDE between 2008 and 2011 was 
technology supported online/virtual learning environments. In figure 6, the following path, 
open education, learning, online, course, and environment demonstrates how these concepts 
are tied to each other and how other peripheral concepts clustered around them. 


Rise of the student centered, technology supported education and data-driven approaches 
(2012-2015) 

Even though some themes and concepts such as information, technology, education, 
learning, are reoccurring in this time period, the final time period was entitled Rise of the student 
centered, technology supported education and data-driven approaches. The major themes were 
students (100%), education (49%), technology (18%), development (11%), data (11%) and time 
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(1%) (Figure 7). According to the analysis of articles published in TOJDE, the first time period 
(2000-2003) was rise of the ICT and e-learning, the second time period (2004-2007) was 
increasing use of educational technologies in distance education, and the third time period 
(2008-2011) was technology supported online/virtual learning environments. As a natural 
consequence of the previous periods, a shift from technology centered research to student 
centered, technology supported education and data-driven approaches emerged in the fourth 
time period (2012-2015). 
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Figure 7: Concept map for the time period between 2012-2015 


As can be seen in the path method-used-data-analysis (Figure 7) increasing the use of 
data-driven approaches has resulted in research that has adopted mainly quantitative 
approaches. This confirms Latchem’s study (2009, in which this tendency was identified, and 
it seems that this tendency continues increasingly. However, as can be tracked in path 
experience-students-online-learning-environment-elearning-ICT, the main focus was student 
centered, technology supported education. When examining the articles published in TOJDE, 
it can be seen that topics such as students’ academic achievements (Unal-Colak and Ozan, 2012; 
Hanbay, 2013; Deniz, Kesan and Izgiol, 2013; Celikoz and Gursoy, 2014), satisfaction (Green 
and Denton, 2012; Calli, Balcikanli, Calli, Cebeci and Seymen, 2013; Goulimaris, 2015; 
Anagnostopoulou, Mavroidis, Giossos and Koutsouba, 2015) and learning styles (Omidvar and 
Tan, 2012; Wu, 2014) were salient. 


If we compare the period referred to as, the rise of the student centered, technology 
supported education and data-driven approach, which emerged in articles published in TOJDE 
between 2012 and 2015, with other periods that emerged in Distance Education journal and 
IRRODL, we can see how TOJDE followed a different path. It was interactive learning, 
MOOCs and OERs (2010-2014) in Distance Education journal and the emergence of MOOCs 
and OER for 2011-2015 in IRRODL (Zawacki-Richter et al., 2017). 
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Overall evaluation and comparison of the themes and concepts 

When examined, it was seen that the main characteristics in all concept maps was the 
information and communication technologies. This indicates that TOJDE has a special focus 
on educational technologies in distance education processes. 


The Turkish Online Journal of Distance Education: TOJDE 
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Figure 8: Comparison of the main themes in TOJDE, IRRODL and Distance Education 
Journals 


If we compare the main characteristics of the periods that emerged in TOJDE, Distance 
Education, and IRRODL (Figure 8), we can see that all three journals reacted to the 
developments that had taken place by the 2000s in the same manner; however, as time 
progresses, we can also see that their focuses shifted to different aspects of the distance 
education field. While TOJDE laid special emphasis on topics such as ICT, educational 
technology, online learning environments and use of data; Distance Education and IRRODL 
shifted their focus to topics such as online learning institutions, interaction, learner support, 
online collaboration and accessibility. One thing that is very striking, this focus became very 
distinct and apparent after the first decade of the new millennium. While articles published in 
TOJDE covered topics related to the use of the data, articles published in Distance Education 
and IRRODL covered topics on new approaches and learning models, such as open educational 
resources and massive open online courses. These three journals represent different regions, 
and the articles originate from different countries. Considering that progress across the globe is 
not at the same pace and the needs of the regions vary according to socio-economic need, and 
probably many other reasons, this is natural and expected. In a broader perspective, it is also 
promising to see that researchers who contribute to these journals do not follow the same trends, 
thus adopting a realistic perspective and contribute to the field by providing knowledge in a 
wide spectrum of research interests. 


Conclusions 


With the purpose of identifying trends and patterns in distance education from the 
perspective of TOJDE, this study examined articles published between 2000 and 2015. The 
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computer-assisted content analysis showed that the rise of the ICT and e-learning (2000-2003), 
the increasing use of educational technologies in distance education (2004-2007), technology 
supported online/virtual learning environments (2008-2011), and the rise of the student 
centered, technology supported education and data-driven approaches (2012-2015) were the 
main themes. 
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Optimizing the Culturally Immersive Peace Corps Experience for Long-Term 
Commitment to Public Service in the Education Sector 


Phillip J. Ward 
University of South Alabama 


Abstract 

This study investigates the academic and professional relationships between Peace Corps 
Volunteers and host-country partners that continue post-service. Peace Corps sends citizens of 
the United States abroad to work with host country partners in different sectors, including 
education. This study will address the impact of partnerships that originated from relationships 
maintained by Volunteers from the education sector and their host-community after the 
obligatory two-year service ended. 

Introduction 


The purpose of this study is to examine the impact of culturally immersive experiences 
and cultural understanding on international academic and professional partnerships. This 
research will include participants from the Returned Peace Corps Volunteer (RPCV) community. 
The study will address motivations for choosing to maintain a working relationship with 
international partners after in-country service has ended. The depth, breadth, mutual benefits, and 
sustainability of a transformative partnership will also be examined. 

The unique culturally-situated learning experiences that Peace Corps provides Volunteers 
in-country sets the foundation for successful integration to host communities. The in-country 
community-based training program includes situated-language instruction, training in language 
teaching methodology, classroom observation and teaching practice, and key area studies 
(history, economics and cultural norms). Skills acquired during pre-service training are further 
enhanced by daily interactions with the host community. PCVs and the host community develop 
a mutual cooperation and learn about each other’s academic expertise and interests. PCVs are 
trained to identify needs of the host institution through formal learning experiences. 

Once the two-year service ends many PCVs maintain relationships with host partners 
through distance mentoring and training, providing access to technological tools and shared 
educational resources. Appropriateness of strategies employed at this stage are implicitly 
established through mutual cultural understanding and capacity awareness. The inclusive nature 
of activities can lead to a more formalized relationship, as is the case of this study’s partnership. 
My motivation to pursue this research is to explore the outcomes of international volunteer 
service for host countries and Returned Peace Corps Volunteers. The philosophy of developing 
mutual cultural understanding, to establish a learning community that provides access to 
technologies essential to learning. 
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Peace Corps 


Peace Corps, an independent agency of the United States (U.S.) federal government, 
sends qualified citizens to communities abroad for a minimum of two years to work with host 
partners in resource- restricted nations. PCVs live in communities where they serve, learn the 
local language of their community, and are trained to identify priorities, strategies, and activities 
to build local capacity in specific sectors, such as agriculture and education (Lough, et al, 2014). 
Members from the host community develop professional and social relationships with PCVs 
through daily interactions. Interpersonal relationships between PCVs and host country partners 
and acceptance of PCVs within the communities lead to improved cross-cultural understanding. 
PCVs who integrate into successfully gain trust and respect within the host community (Hall, 
2007). Among host country partners, frequent contact with PCVs leads to a greater 
understanding of Americans. 

RPCVs often make concentrated efforts to maintain relationships with their host 
communities to create sustainable change that lasts after in-country service. The culturally 
immersive experience and cultural understanding that results from Peace Corps experience is a 
conducive element for creating transformative international academic and professional 
partnerships. The outcomes of the partnership are transformative as opposed to transactional. 
Transformative partnerships are multi-dimensional, mutually beneficial, and more likely to be 
sustained over time. 


USA-IFNTUOG Partnership 


I served as PCV in a Ukraine from 2010-2012. I was assigned to work with a Ukrainian 
counterpart, who served as a linguistic interpreter and liaison to the host community for two 
years. The counterpart also acted an agent of professional integration and cross-cultural mentor 
at the Ivano-Frankivsk National Technical University of Oil and Gas (IFNTUOG) in Ivano- 
Frankivsk, Ukraine. 

After the end of in-country service 2012, my Ukrainian counterpart and I maintained a 
collaborative academic relationship. From 2014 to 2018 the relationship was informal in nature. 
IFNTUOG students were introduced to online learning as part of the initial stages of 
collaboration. My colleagues from the University of South Alabama (USA) and I trained my 
counterpart in online course design using educational technologies provided by our institution. 
This experience revealed a mutual interest in instructional design, online course development, 
and student engagement. 

In 2018, the informal collaborative relationship was elevated to that of a formal 
partnership, with the signing of a Memorandum of Understanding between the both partner 
universities. This official action recognized the importance of the relationship, ensured 
institutional commitment, and gave my former Ukrainian counterpart a higher status as an 
international partner. 

Since the formal partnership was established, our work has expanded. Our Ukrainian partner 
hosted the first webinar in the series on the topic of developing verbal English skills outside the 
classroom using TEDx techniques. This was a presentation at the Conference on Teaching and 
Learning, hosted by my university. The conference was virtual, so it was also attended by her 
Ukrainian colleagues. She was then asked to train other instructors in her department to use web 
conferencing technology (i.e. Zoom). This was of major importance when Ukrainian campuses 
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closed due to COVID-19. The technical skills she for our department’s remote professional 
development workshops and one-on-one training had a positive impact in her educational 
environment. 

We engage her students in partnership activities to develop global competencies. Student- 
initiated classroom participation and self-regulated learning is not presently characteristic of 
Ukrainian higher education. The partnership gives Ukrainian students access educational 
technologies that reflect the cultural learning practices of that country. The partnership integrates 
online learning experiences with traditional face-to-face education to promote a more learner- 
centered curriculum at the partner institution (Loo, et al, 2019). 

My motivation to pursue this research to explore the outcomes of international volunteer 
service for host countries partners and Returned Peace Corps Volunteers. The current research 
that exists on international service does not provide evidence of impact, rather the majority of 
current research in this area is based on case and cross-sectional studies (Devereux, 2008; Lough, 
et al, 2014). 

The following research questions will be explored in this study. 

e What are some of the changes in values and beliefs that individuals experience during 

Peace Corps service? 

e How did cross-cultural experiences with individuals from the host country affect these 

changes? 

e Did the experience result in a commitment to the education sector or related service- 

oriented career paths? 

e What types of relationships do RPCVs maintain with host country individuals and 

organizations? 


Method 
Procedure 


I chose to use mixed-method approach for this study. The research is conducted in two 
phases. Phase 1 (P1), described in this paper, used an online questionnaire to collect quantitative 
data. The quantitative data collected in the questionnaire identified individuals who meet the 
inclusion criteria required to participate in Phase 2 (P2). I used descriptive statistical analysis to 
examine the 32 participants on the quantitative variables. Because of the qualitative focus of this 
study and the small number of cases, I did not use statistical significance testing. 

Starting in P2, qualitative data will be gathered through in-depth interviews conducted 
over Zoom. A letter of Informed Consent will be given to each participant prior to each 
interview. During this phase, the perspectives of each participant will be revealed through 
qualitative methods using Strauss and Corbin’s (2015) approach to data analysis. The constant 
comparative method will be used to develop concepts from the data through continual data 
collection, coding, and analysis. 


P1 Participants 
Thirty-two individuals participated in P1 this study. All were Returned Peace Corps 
Volunteers who satisfactorily completed two years of Volunteer service in a foreign country or 


completed their Volunteer service with an early Close of Service. Twenty participants were 
female, eleven were male, one self-described as non-binary, and three did not indicate their 
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gender. Twenty-nine participants indicated that they served only once, in one host country. Two 
participants served twice, in a different host country each time. One participant served twice, 
each time in the same country. The earliest starting year of service among the participants was 
1973. The most recent Close of Service year reported was 2020. The participants’ primary 
assignment sectors included agriculture, community economic development, education, 
organizational development, health, and youth in development. 


Findings 


All 32 participants responded to which geographic region of the world they served. The 
responses indicated six different regions. The majority served in Eastern Europe and Central 
Asia (44%). The second region with the where participants served was Africa (22%). Other 
participants indicated serving in Eastern Asia (16%), Central America (13%), Caribbean (3%), 
and South America (3%). 

Ukraine had the highest concentration of participants (34%). Three participants (9%) 
served in Tanzania. Guatemala and Armenia each hosted two of participants. The other countries 
each hosted one participant. Two participants indicated that they served twice, one of them 
served in Mauritania after serving in Chad and the other served in Comoros after serving in 
Kyrgyzstan. One participant responded that they served twice and indicated that they served in 
Ukraine both times. See Table 1 for countries represented in this study. Table 2 depicts the host 
countries of those participants that served in Peace Corps more than once. 


Table 1 
Host Countries of Participants 


Country Number of Participants [Percent 
Ukraine 11 B4% 
Tanzania B 9% 
Guatemala 2 6% 
Armenia 2 6% 
Belize 1 B% 
Chad 1 B% 
Ethiopia 1 B% 
Jamaica 1 B% 
Mongolia 1 B% 
Nepal 1 B% 
Philippines 1 B% 
Romania 1 B% 
Ghana 1 B% 
Burkina Faso 1 B% 
Peru 1 B% 
Kyrgyzstan 1 B% 
Georgia 1 B% 
Honduras 1 B% 
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Table 2 
Host Countries of Participants Who Served Than Once 


Country Number of Participants Percent 
Only Served in Once 29 91% 
Mauritania 1 B% 
Ukraine 1 B% 
Comoros 1 B% 


Overall, 13 of the 32 respondents worked in the education sector for their primary 
assignment. This includes secondary school, higher education, math education, teachers of 
English as a foreign language. The participants indicated their level of agreement with several 
aspects of their Peace Corps experience. Overall, 30 participants strongly agreed that the 
experience was rewarding and two simply agreed. The participants were asked to share their 
secondary community projects. See Table 3 for a self-description of secondary community 
projects that the participants were involved in during their service. 


Table 3 
Primary Assignment Sector and Host Country of Participants 


Country and Service Sector 


Armenia Education 
Belize Served in Two Sectors (Health and Youth in Development) 
Chad Served in Two Sectors (Education and Community Development) 
Ethiopia Education 
Guatemala Agriculture 

Health 
Jamaica Education 
Mongolia Education 
Nepal Education 
Philippines Health 
Romania Organizational Development 
Tanzania Education 
Ukraine Education 

Community Economic Development 
Ghana Education 
Burkina Faso Health 
Peru Served in Two Sectors (Education and Community Development) 
Kyrgyzstan Education 
Georgia Education 
Honduras Education 


Overall, 13 of the 32 respondents worked in the education sector for their primary 
assignment. This includes secondary school, higher education, math education, teachers of 
English as a foreign language. The participants indicated their level of agreement with several 
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aspects of their Peace Corps experience. Overall, 30 participants strongly agreed that the 
experience was rewarding and two simply agreed. The participants were asked to share their 
secondary community projects. See Table 5 for a self-description of secondary community 
projects that the participants were involved in during their service. 


Table 5 


Self-Description of Secondary Community Projects 


Secondary Community Projects 


Armenia English clubs at the windows on America library, baseball camps, leadership 
camps 
English clubs; English poetry recitation club; English camps 

Belize English teaching, youth soccer club, gardening club 

Chad Women's credit union Building wells for farmers (non-potable water) Scholars 
bowls for middle school maintenance (project started by prior PCV): - Garden - 
Computer room 

Ethiopia English club, Drama club, Grassroots Soccer, Gender & Development Club, Girls 
Club, HIV support group, entrepreneurship program for HIV positive women and 
children, GLOW cam; hygiene and awareness program for the prison population. 

Guatemala [English Clubs 
IESL, WID (nutrition, French Intensive gardening) 

Jamaica Environmental science club for teens 

Mongolia English lessons for teachers 

Nepal Taught primary-level science (primary assignment was English) 


Philippines _Diversity camps, Leadership camps 


Romania GLOW/TOBY (gender) camps; environmental activities 

Tanzania English book club, English movie night, weekly English language club 
Grassroot Soccer 
IHIV-aids awareness 

Ukraine English and Journalism club, summer camps, community clean-ups 
English Camps for Ukrainian teachers of English. I also raised money to purchase 
a computer for my school. 
English Camps, English Clubs 
English camps, English clubs, HIV/AIDS awareness/prevention camps, human 
trafficking awareness/prevention camps, teacher training camps, drama camps 
English camps, English movie clubs, GLOW seminars for girls 
English club for youth; English club for adults; helping friend with her after- 
school students practicing their English lessons. 
English Club University English language classes Plast (youth scouting 
organization) activities 
English summer camp and English club 
IEnglish/youth leadership development/environmental/HIV AIDS camps, English 
clubs, after school tutoring 

Ghana English Club 
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Burkina Faso [English classes, cooking classes, cognitive stimulation class for senior citizens, 
girls soccer club, movie club, exercise clubs. 


Kyrgyzstan [English clubs, environmental camps, competitions & field days 


Georgia Nutrition awareness, feeding the male nourished, supporting women’s income 
generating projects of clay, seeds, clothing, and corn husks designs. 


Next Steps 


The data collected for this exploratory study will be used to develop questions for the in- 
depth interviews that I will conduct in Phase 2. The interview questions will also be based on 
data collected on existing literature on Returned Peace Corps Volunteers and the cultural, social, 
academic post-service outcomes. After the interviews are conducted, each case will be examined 
in totality. The different cases will be compared for patterns that emerge and differences that 
exist. 
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Abstract 


This paper presents an emerging model of social justice education that has been 
implemented in a rural K-12 school system in the Southeastern US. The model was developed 
over a five-year period through a research practice partnership between a state university (e.g., 
college of education) and the local community. Using a DBR approach, the research team 
worked closely with teachers and students to take a process of historical inquiry and embed it 
into the K-12 school system. The result of this effort is an emerging model of learning that 
attends to different components of social justice education, including the doing of history, the 
juxtaposing of various narratives, and the development of critical consciousness. These 
components are then illustrated through a case study that focuses on the interactions between the 
components and how those interactions supported students’ meaning-making. Implications for 
the design and study of learning in the context of social justice are discussed. 


Introduction 


This project involved creating bridges between a state university and K-12 students in 
rural and economically-disadvantaged Putnam County, Georgia. One goal was to open a 
discussion of Putnam’s past among those who grew up there decades ago and the students who 
make up its future. Our hope was that building a model of K-12 education to support and engage 
in these dialogues would develop students’ critical awareness of who they are and where they 
come from, and ultimately empower them to realize their potential in the world as fully as 
possible. Over the course of a five-year research-practice partnership, we, the authors of this 
paper, have worked together to construct a model of K-12 education in which both teachers and 
students listen to, uncover, and preserve the stories of the community members while using those 
stories as a tool for developing critical perspectives about the history of Putnam county. 
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With that in mind, this paper begins with an introduction to the research-practice 
partnership and the community in which the partnership was established. After a brief summary 
of the partnership, we present an emerging model of learning that draws on our experiences over 
the past five years. The model, which was built through a design-based research approach, is 
then illustrated through a single case in which two students engaged in the “doing” of history, the 
juxtaposition of narratives, and the practice of critical consciousness. The individual components 
of the model as well as the importance of their intersectionality are then discussed. 


Research-Practice Partnership (RPP) and Design-based Research (DBR) 


The RPP began with a broad goal of building the in-school experience of K-12 students 
around their local community, where the community served as a place for teachers and students 
to both meet standards and engage in a democratic society. To guide our work, we drew from 
design-based implementation research (DBIR) (Penuel, Fishman, Cheng, & Sabelli, 2011), 
which embraces an iterative process of developing, testing, improving, and retesting a research- 
driven educational intervention through deep collaboration with local contexts (see also Cobb et 
al., 2003). DBIR emphasizes the value of co-design, meaning researchers and local stakeholders 
(e.g., administrators, teachers, students) work collaboratively to shape and accomplish the 
driving goal of the project. Examining a well-designed intervention over time and in a local 
context leads to the formation of research-based learning principles and practices that advance 
theory while having relevance in an applied context (Penuel et al., 2011). 

The RPP consisted of several entities, including members of the Wilson Center for 
Humanities & Arts and the College of Education at the University of Georgia, and the public- 
school system in Putnam County, Georgia. Putnam County Charter School System is a PreK-12 
system that serves approximately 3,500 students (44% white, 41% black, and 11% Hispanic). 
The percentage of families living below the poverty line is between 15 and 30%, while 79% are 
classified as economically disadvantaged. 

The rich and complex cultural history of Putnam County played a significant role in our 
work as a partnership. Both Joel Chandler Harris (1845-1908) and Alice Walker (b. 1944) were 
born and raised there, and the county figures prominently in their literary works. While Harris’s 
19th-century Uncle Remus tales focus on the time before Putnam’s enslaved residents were 
freed, Walker’s poetry and prose of the 1970s and 80s, such as The Third Life of Grange 
Copeland and The Color Purple, embodies the repercussions of the long period of racial and 
economic inequality and injustice that followed. Walker’s work has been celebrated for creating 
a literature about those who lived ordinary and hard lives in the segregated South, but, drawn out 
of her own experiences, it also provides a critical lens for examining the specific history of the 
place she grew up both before and during the Civil Rights era. 

Putnam is in many ways like most rural communities across the Deep South, including 
that many of its K-12 students are descended from families that have been there for multiple 
generations. For example, many Putnam residents in their 70s and 80s grew up with Alice 
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Walker and her siblings and most residents over 60 have clear memories of segregation and the 
struggle for civil rights. These older generations, who lived through what Walker later wrote 
about, have their own stories to tell of the hardships, setbacks, victories, and changes that have 
transformed the place of their childhoods into the one their grandchildren and great- 
grandchildren are now growing up. The purpose of the project was to create opportunities for K- 
12 students to engage with those community members and their stories, to open a discussion of 
Putnam’s hard past between those who grew up there decades ago and the students who make up 
its future. Our overall goal was to work with teachers and students to capture those stories and 
use them as both a context for, and document of, the history of Putnam county while also 
achieving the performance standards to which they are held accountable. The question guiding 
this research was: What design perspectives/principles were most salient in this project, and 
what did they look like in practice? 


An emerging model 

Through a reflexive design practice, we situated our work within our partnerships with 
teachers and students in order to center and value the different ways of knowing and knowledge 
that each partner brought. This reflexive practice included exploring how to navigate the tensions 
that arose around long-standing racial and economic injustices in the community, and how to 
support teachers and students in exploring those tensions while being sensitive to the historical 
context and people in a community as well as attending to our own identities as white scholars. 
There were no clear-cut answers to these questions. Instead, our work as an RPP focused on 
continuously developing our own understanding of the design through our interactions with each 
other, the teachers, the students, and members of the Putnam community. This has resulted in an 
emerging model of social justice education that consists of three components (see Fig. 1): the 
“doing” of history, critical consciousness, and the juxtaposition of narrative. We call the model 
“emerging” for the simple reason that it has been developed and continues to develop through 
our relationship with each other and the other participants (see Lawton et al., 2020; Walters, 
Kopcha, & Lawton, 2020). That relationship is not static; it grows and shifts each year as new 
challenges develop and accomplishments are realized. 
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Figure 1: An Emerging Model for Social Justice Education. 


The doing of history 

The doing of history is a phrase used in social studies education that refers to engaging 
students in the work of real-world historians in the classroom (Levstik & Barton, 2011). From 
this perspective, history is viewed as something that is constructed by people from the pieces of 
the past that remain today (Wineburg, 2001). This work is always incomplete and subjective -- as 
new evidence comes to light, new perspectives on history can be constructed (Rtisen, 2005). 
Thus, the primary activity in the doing of history is making arguments based on our analysis of 
the evidence at hand. By creating their own evidence-based arguments, students learn that 
history is not static, i.e., the “true” presentation of facts in a textbook, but a process of human 
inquiry into the past driven by current-day interests and questions (Buehl & Alexander, 2006; 
Riisen, 2005). 

In the context of social justice education, many scholars have introduced critical 
perspectives to the doing of history. This takes the methods of historical inquiry and layers an 
additional focus on issues of social justice and equity (Parkhouse, 2018). Activities such as 
creating narratives that center untold stories and marginalized people (Salinas & Alarcon, 2016), 
exploring alternate perspectives on dominant and/or traditional narratives (Freedman, 2015; 
Salinas & Alarcon, 2016), and reflecting on their own perspectives and assumptions (Lim, 2010; 
Parkhouse, 2018) can support students in critical historical inquiry. 


Critical consciousness 

Critical consciousness is a philosophical perspective that interrogates long-standing 
traditions and practices in society. Engaging in critical consciousness means observing and 
identifying the social and political structures that shape the world, and developing an awareness 
of how those structures affect different groups of people differently. The notion of critical 
consciousness is rooted in the work of Brazilian educator Paulo Freire (1973; 1974) who worked 
with illiterate, impoverished adults in Brazil to understand the way oppression becomes 
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operationalized in society and use this knowledge to create change. For Freire and others who 
have adopted this perspective (e.g., Cammarota & Fine, 2008; Paris, 2012; Souto-Manning, 
2010), there is a common emphasis on action (i.e., critical action, social change, participation), 
self- and socio-political efficacy, and critical reflection. Related practices in K-12 include 
Ladson-Billings’ (1995) Culturally Responsive Pedagogy (CRP) and Paris’ (2012) Culturally- 
Sustaining Pedagogy. These reposition students as ‘knowers’ in the classroom, attempting to 
disrupt their assumptions about the source and nature of our knowledge. All students’ unique 
perspectives, experiences, and cultures play a role in the way that students see themselves as 
agents of knowledge and have space to engage in critical dialogue (Taylor, 2019). 


Juxtaposing narratives 

The act of juxtaposing two ideas or concepts, such as narratives, is rooted in pedagogical 
strategies that promote alternative perspective taking (Jonassen, 2011) and cognitive flexibility 
(Spiro et al., 2011). The general idea is that juxtaposing narratives about the same person, time, 
and/or place creates an opportunity for students to observe what is similar and, more importantly, 
where differences in those narratives arise. Those differences are often rooted in long-standing 
political and social issues that shape the structure and nature of each narrative. Thus, juxtaposing 
narratives is a powerful way of supporting both the doing of critical historical inquiry and the 
development of critical consciousness. This work allows for students to engage with multiple 
perspectives, reveal normative assumptions, and provides spaces of action for critical dialogue 
and identity formation. 

Methods 

Broadly, the project has grown over the past five years to include 25 teachers and their 
students as part of the RPP. In this paper, we present a single descriptive case study (Yin, 2017) 
of two high school students engaged in a semester-long critical historical inquiry project. The 
analysis focuses on the meaning-making processes that were evident in their work products, 
particularly as related to the doing of history, critical consciousness, and juxtaposing narratives. 
The two students serve as embedded units that represent the phenomena of the meaning-making 
process (Baxter & Jack, 2008; Yin, 2017). 


Participants 

Participants included two students enrolled in an inclusion-level 10th grade U.S. History 
course. Raymond (all student names are pseudonyms) is an African American male whose 
family is from Putnam. Ashley is a white female student, also from Putnam. Their teacher, a 
white male, was among the first members of the project; at the time of this study, he had been 
teaching history for about 12 years. 


Design and analysis 


This case focused on the students’ meaning-making process during historical inquiry and 
was selected because it illustrates the way different components of the model supported the 
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project goals. Student-produced artifacts, including worksheets, archival research, and written 
and audio reflections, were analyzed using a social semiotic approach to multimodality. The 
social semiotic approach centers on a sign-maker’s (in this case the student’s) meaning-making 
process as presented through the choice, use, and integration of various modes (e.g., color, text, 
tone, gestures) (Jewitt et. al, 2016; Kress, 2010). According to this theory, meaning is 
communicated through the interaction between the modes rather than any singular mode (Kress, 
2010). 

Consistent with a social semiotic multimodal approach, our analysis included a focus on 
how multimodal artifacts related with the social, cultural, and historical context. This was 
ethnographic in nature, in that we collected and analyzed documents related to these contexts in 
addition to student work products (Coffey, 2018). We engaged in data collection and analysis 
throughout the project, continually revisiting our data to inform future collection and analysis 
choices in a non-linear process (Coffey, 2018; Madison, 2020). This process included a close and 
detailed reading of all the data to construct a holistic understanding of the case (Coffey, 2018). 
We then situated the students’ historical inquiry work and reflections within the social, cultural, 
and historical context of Putnam County. 


Meaning-making during critical historical inquiry 

Over the course of a semester, the two students in our case study engaged in a high 
school U.S. history course. The curricular unit addressed here, Understanding our community's 
roots: A beginning lesson plan setting the stage for connecting local and national history, was 
built to meet part of the Georgia standards for Reconstruction and the era known as “Jim Crow.” 
Students explored the political, economic, and social histories of the period by engaging in 
lessons and projects that focused on historical inquiry in the local community. Through 
examples, modeling, and practice, students in the course learned to locate and analyze primary 
sources, collect their own primary sources (e.g., census records, genealogical data, oral histories 
with local elders, etc.) and conduct historical research in their community. The course was 
structured to connect students to their community through place-based lessons, research, and 
reflection. 


The doing of history 

Throughout the semester, students engaged in collecting, assessing, and interpreting 
historical source material to construct their own understandings of their community’s past. 
Students gradually took on the role of historian as they completed activities that scaffolded their 
inquiry process. The inquiry began as the students and teachers wanted to know more about the 
families who attended a local Baptist church found in photos of the community taken in 1941. 
Because the church still stands today, the class began working with 1940 U.S. census documents 
to collect information about the families who lived there at the time and may have visited the 
church regularly. To support the activity, they also made multiple visits to the church and the 
cemetery located on the church grounds. Using census records and the information available on 
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tombstones, as well as ongoing conversations with the elder deacon, who has been a member of 
the church since his childhood in the 1940s, they then constructed family trees and composed 
short narratives about their inquiries. The work of interpreting primary sources is critical to 
historical inquiry, as is using evidence from these sources to construct historical narratives 
(Seixas, 2015; Wineburg, 2001). 


Juxtaposing narratives 

In both our cases, students began by engaging with existing historical narratives before 
moving towards constructing their own. This process of juxtaposition was informed not only by 
their research, where narratives were explicitly taught and constructed, but also by allowing 
space for their lived experiences to enter into analysis. As teenagers growing up in a rural and 
economically-disadvantaged Deep South community, both students were shaped in part by 
various metanarratives about race, class, and rural life. 

These existing narratives were challenged by the students’ research into the community -- 
specifically, they began inquiring about the only land-owning African American family in the 
area at the time of the 1940 census. The family’s land-owning status created an opportunity to 
juxtapose various existing narratives. The dominant narrative was that African Americans at the 
time typically occupied low social and economic statuses, particularly in a rural community. The 
story of this land-owning family, then, offered an opportunity to understand a different narrative, 
one in which an African American family held a higher status than expected. 

Ashley highlighted the family’s economic status in her narrative. She included the 
amount of money the family’s home was worth and noted how it was “their own home,” 
[emphasis added]. She then situated these facts in a historical context. Her attention to the fact 
that, “They are an African American family,” suggests that she is aware of how the family’s 
narrative runs counter to the existing narrative. The inclusion of the family’s racial identity 
reveals both her awareness of the dominant narrative of the intersection of race and economic 
status and her understanding of the historical significance of this family as a counternarrative. 

Raymond also engaged with these contrasting perspectives of his community, but in a 
different way than Ashley. He addressed negative metanarratives of rural communities more 
broadly, stating, “For years people have said rude things about Putnam County...things like ‘its 
lame’ and there is ‘nothing to do’ there.” He found these existing negative perspectives 
contrasted his own gut-feeling that Putnam “was something special.” As he uncovered and 
constructed a historical understanding of his county, he realized that its history was the 
“something special” he had been feeling. He stated, “if people took time to learn [Putnam’s] 
history they’ ll see that it's special as well.” The juxtaposing of narratives helped Raymond see 
how rural communities are often perceived as stagnant places with “nothing to do,” yet new 
evidence can help reveal such communities as dynamic, diverse, and important. 


335 


Critical consciousness 

The meaning-making supported during these lessons went beyond a disciplinary focus or 
understanding. The students made personal connections to the historical narratives they learned 
and constructed, exploring the ways that power structures influence our understanding of both 
past and present. At one point, Ashley expressed concern over families having to experience 
things “they did not deserve” as a result of segregation, as well as acknowledged people’s 
resilience in getting through difficult times. These empathetic connections enabled her to see the 
human connection between past and present peoples, as well as to make sense of current-day 
emotions and actions rooted in past oppression: “There will always be this really dark part of 
history that lives in the back of some people's minds. There will always be a sense of hatred that 
lives in some people’s hearts.” Ashley’s reflection suggests that she felt that what happened in 
the past was unfair. She also recognized how the story of Putnam is actually reflected through 
multiple stories, and that there are “unspoken” and unheard stories that represent the “heart of 
Putnam.” She concludes her reflection looking towards action, stating these stories must continue 
to be told. 

Raymond’s reflection similarly wrestled with ideas of power, stories, and action. He 
recognized the power of historical knowledge, particularly in the stories of those traditionally 
marginalized. He was upset by his newly-acquired knowledge of the painful history of 
enslavement, segregation, and reconstruction in the county, especially how this history and its 
legacy impacted both people in the past and the present. Moreover, he felt angry because the 
stories represented a missing piece in his understanding of himself “Learning this story so late 
into my life angers me,” he wrote, adding, “It hurts to learn this now.” Like Ashley, he also 
expressed a desire to take action and stated that he would use the information he learned and 
[do] something” with it, although, “learning it sooner would have been very useful.” 


Discussion 

Each component of our model represents an important component of learning, with an 
established body of literature in its own right. In the case above, we have tried to illustrate how 
each of those components was embodied in the learning activities that the teachers designed, as 
evidenced through the work that students produced. For example, the doing of history was 
supported through hands-on field work by the students, who visited a church whose history 
reaches back to the family they were research (ca. 1940) and beyond. There, they explored the 
names of those buried at the church and used various archives (e.g., census data) to learn more 
about the community. This supported them in juxtaposing existing narratives about the 
community with other less dominant narratives and engaging in reflective practices that 
supported a critical position on themselves and their ancestors (i.e., critical consciousness). This 
study lends support to the importance of activities associated with critical historical inquiry, 
historical narrative, and critical pedagogy in the K-12 classroom (Ladson-Billings, 1995; Paris, 
2012). 
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‘hod what does it mean to you te learn about ts era of you" 


Figure 2: (Clockwise from top left) Page from 1940 U.S. Census for Putnam County, student-created 
census worksheet, family tree, and reflection produced as part of an on-site visit to a local church. 


While we have presented these three components of our model as separate and distinct 
entities in our design work, we have an emerging sense that it is the intersection of these three 
components that is as, if not more, important than each component individually. Simply put, the 
sum of our model ends up being greater than its parts. While each component introduces a 
different way of thinking and knowing to the project, it is their integration that supports our 
participants in developing what we are calling a critical-dialectical-historical perspective. This 
perspective is not just a reflection of the overall design of the project, but also a mechanism 
through which all participants -- the researchers, teachers, students, community members -- 
engage in the process challenging notions of “truth” that exist around the historical and social 
narratives and artifacts that reflect that truth. 

A critical-dialectical-historical perspective demands that one questions what we know, 
collectively and individually, and how we have come to know it -- whether we are looking back 
to the past, making sense of the present, or looking forward to the future. It means taking the 
evidence we have, that is part of the historical record, and ascertaining the social, political, and 
economic systems that have contributed to that record. Because ultimately, those systems have 
produced a framework that determines whose stories are permitted to be left behind, and 
therefore what narratives are available as meaning-making resources in the present (Wineburg, 
2001). 

In the case we have presented, the C-D-H perspective emerged as one in which students 
opened conversations about issues not normally addressed in a K-12 classroom. Not only did 
students and teachers discuss issues of injustice and inequity, they did so in ways that directly 
connected to students’ out-of-school lives and the broader community. The long-standing 
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patterns of injustice/inequity from the past were no longer abstract. The students described their 
findings with phrases like “a sense of hatred,” “it hurts,” and “dark part of history” -- these 
suggest how they connected with the themes of injustice from their own community by going 
beyond dates and facts. Their construction of historical narrative came with a sense of empathy 
and presence within their community, creating an opportunity for them to be more critical of the 
structures that contributed to those patterns. Comparing what had once happened to their own 
lived understanding of the present compelled them to engage more deeply with their sense of 
justice (e.g., “The opportunity to ...give this family and the Harmony community the recognition 
that is long overdue”’) and speak up about the injustices they saw (e.g., “This kind of history is a 
serious topic and a fragile thing to tell’). It was the tension between the past and present 
narratives (i.e., dialectic) that created an opportunity for deeper engagement with developing a 
critical perspective on the history of the community. 

Historical perspective taking and identifying continuity and change are two of the major 
components of the historical inquiry process (Seixas, 2015). In our project, students engaged in 
both through their juxtaposition of narratives while engaged in the “doing” of history. The doing 
of history opened up space for student-centered, place-based inquiry. As implemented in our 
RPP, this inquiry involved community engagement and a focus on historically-marginalized 
peoples’ stories. Juxtaposing narratives occurred when students compared stories about the past 
as told from multiple perspectives, as well as when they compared their previous understandings 
to those gained through inquiry. With teacher support, students developed an understanding of 
how narratives provide insight into social and political structures and the ways these structures 
impact individuals and communities across time. 
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Abstract 


Amid the COVID-19 outbreak in the United States, PK-12 teachers quickly transitioned from 
their face-to-face classrooms to an e-learning environment due to the shutdown of schools. To 
better understand teachers’ experiences with this transition, a 31-question survey was developed 
and sent to in-service PK-12 teachers enrolled in a small, private liberal arts Lutheran university 
in the Chicagoland area. Survey responses (n=109) demonstrated needs relative to classroom 
management, e-learning pedagogy, technology access, and clearer policies for e-learning moving 
forward. Next steps for continued research into this topic and the ways to support those who are 
teaching children in today’s online classrooms are discussed. 


Keywords: PK-12 teachers, e-learning, educational technology, COVID-19 


Introduction 


During the coronavirus (COVID-19) outbreak in the United States, PK-12 teachers had to 
quickly transition from their face-to-face classrooms to an e-learning environment due to the 
shutdown of schools and shelter-in-place orders across states in the spring of 2020. Many 
teachers were new to the e-learning landscape and found that they suddenly needed to support 
students and parents who were acclimating to learning at home. Additionally, educational 
policymakers were scurrying to put policies in place that would support teachers and their 
students while considering various limitations of e-learning. 

As teachers were suddenly switching to e-learning during the shutdowns due to COVID, 
the researchers sought to better understand the strengths and challenges with this transition to 
assist other teachers and administrators in adapting to emergency e-learning situations in the 
future. This article provides an introduction to the research study, a look at the profile of survey 
respondents, and the patterns and themes identified from the data collected. The conclusion 
provides information about what this means for educators as well as ideas for the next steps in 
this research. 


Theoretical Framework 


There are several theoretical perspectives that provide insight into the function and design of 
online learning experiences. The “community of inquiry” model developed by Garrison, 
Anderson, and Archer (2000) is particularly helpful for this purpose. In their model, it is claimed 
that three “presences” combine to shape a student’s online educational experience: cognitive 
presence, social presence, and teaching presence. Cognitive presence is the extent to which 
participants can learn through sustained communication with other participants. Social presence 
is the ability of participants to connect socially and emotionally in the environment. Teaching 
presence is the ability of the teacher to effectively design and deliver instruction. 
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The “community of inquiry” model provides a helpful lens for analyzing the responses 
teachers provided about the constraints, resources, technology tools, successes, and failures, 
experienced during the move to e-Learning because of the COVID-19 pandemic. 


Method 


A 31-question survey was developed by the researchers about teachers’ backgrounds and settings 
in e-learning, including their instructional practices, the technology used, and recommendations 
moving forward based on their experiences transitioning to e-learning during school shutdowns 
due to COVID-19. 

The survey also included questions informed by the community of inquiry (COI) model, 
which suggests that a successful online learning environment must incorporate teaching 
presence, social presence, and cognitive presence (Garrison, Anderson, & Archer, 2000). The 
COI model links to Kolb’s (1984) experiential learning cycle, which focuses on a student’s real- 
life learning context, particularly in the form of primary social relationships as contexts for 
experiential learning, as well as the input of the teacher (teacher presence) and content (cognitive 
presence). 

The survey was sent to in-service PK-12 teachers enrolled in graduate-level teacher 
education programs at a small, private Lutheran university just outside of Chicago. The 
university has offered teacher education since the late 1800s, and teachers continue to make up a 
significant portion of its student body. The majority of teachers in these programs teach in 
private and public school districts in the Chicagoland area; a smaller portion of teachers are from 
other parts of Illinois and out-of-state. The researchers are members of the faculty at this 
institution, so the sample was one of convenience but also made sense considering the need for 
quick and easy access to a teacher population during the emergency COVID-19 situation. 

With IRB approval granted, the survey was developed in Qualtrics and distributed via a 
link in an announcement in the teachers’ courses in the university’s learning management system 
one month after Illinois Governor Pritzker’s shelter-in-place order on March 21, 2020. 
Participants gave their informed consent, were assured anonymity, and were assured that their 
participation would not affect their grades. The survey consisted of multiple-choice, fill-in-the- 
blank, and Likert scale questions related to teachers’ school setting, resources, understanding and 
implementation of policies, instructional strategies, and overall experience using educational 
technology in the transition to e-learning during the COVID-19 pandemic. 

Survey responses (n=109) were collected from April 24 to June 26, 2020. The researchers 
conducted a descriptive analysis of the quantitative data and thematic coding of the qualitative 
data using Qualtrics analysis software. 


Survey Respondent Profile 


Responses were received from teachers from a variety of settings. Of the 109 respondents, 38 
taught in K-4 settings, 41 taught in grades 5-8 settings, and 24 taught in high school settings. The 
greatest number of respondents had 4 to 6 years of teaching experience. But there were many 
teachers with even more teaching experience. So data was collected from a large number of very 
experienced veteran teachers. 
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On the other hand, the vast majority of teachers had little or no experience teaching 
students fully online. In fact, only four respondents said they had taught their students fully 
online as a part of their school’s regular curriculum. 


Survey Results 


The results are grouped into four categories: (1) preparation and professional development 
received, (2) changes in practice, (3) technology used, and (4) recommendations from 
respondents. 


Preparation and Professional Development Received 


Teachers prepared for e-learning in three major ways: professional development, preparing, 
organizing, and sharing content, and setting up communication and record keeping. The list 
below provides examples teachers gave for each of these categories. 


Professional development: 
e planning day at school before shutdown 
e some training/professional development for technology at school and district level: PLC, 
coaches 
e divided work and collaborated with teacher teams 
e research online resources, Teachers Pay Teachers, etc. for making lessons and 
assignments digital 
Preparing, organizing, posting content: 
e consider what activities and technology students were already familiar with 
e make checklists, daily schedules, routines 
e wrote out instructions for classroom assignments 
e recorded instructional videos and built slideshows 
e prepared to teach with print as well as digital resources; convert print to digital 
e posted digital content to LMS and district websites 
e not much to do if already using an e-learning platform or had a blended or flipped 
classroom 
Setting up communications and record keeping: 
e set up video conferencing software 
e create email/phone lists 
e set up virtual office hours 
e send expectations to parents and students 
e set up log for tracking work and participation 


Almost all respondents participated in some type of professional development during the initial 
shutdown of schools, whether offered by their school, through teaching teams, or self-directed 
learning. To prepare for the school response to the Covid-19 pandemic, 75% of respondents 
indicated that they had received at least some form of e-Learning training before moving to a 
fully online environment (see Figure 1). Of those who received some combination of training, 
45% participated in formal professional development meetings, 21% received video or other 
multimedia training resources, and 26% received written materials. About two thirds of the 


342 


respondents said they had access to someone at their school who was knowledgeable of the 
technology integration, strategies, and pedagogical approaches to fully online teaching most of 
the time or always. 


Figure 1 


Prior e-Learning Training 


45% 
26% 25% 
21% 
20% 
10% 
0% 


Professional Video and Other Written Guidefor No Training Indicated 
Development Multimedia Training Teachers or Other 
Meetings Written Resources 


Teachers then began to prepare, organize, and share content, such as by making daily 
checklists, creating instructional videos, and posting content to their e-learning platform or 
district website. However, some respondents noted that if their school was already 1:1 (e.g. each 
student uses a device on loan from the school) or if they were already using apps for learning, 
they didn’t have to prepare much. 

Lastly, they set up communications and ways to keep records of students’ experiences 
with e-learning, such as video conferencing software, virtual office hours, and logs for tracking 
attendance and participation. 


Changes in Practice 


The requirements teachers received from their districts for online learning varied, but 38% 
reported needing to do daily briefings for students. 25% took no attendance, and 50% were told 
not to give grades. With the move to online, 78% of respondents said they communicated with 
their students multiple times each week, with 50% saying they did so every day. Communication 
frequency with parents saw an increase for 67% of respondents, while only 6% reported a 
decrease in the frequency (compared to before the shutdown). 

With the move to fully online learning, it was expected that a teacher’s sense of being 
socially connected to students (or ’social presence”) would get worse. Not surprisingly, 76% of 
teachers who responded to this specific question, felt their sense of social connection worsened 
during the shutdown. A teacher’s ability to effectively design and deliver instruction (or 
teaching presence’’) also saw a negative change, with 70% of teachers who responded to this 
specific question, indicating that things got worse (see Figure 2). Finally, the ability to promote 
student learning through reflection and discussion (or ’cognitive presence”) saw a negative 
change, with 74% of teachers who responded to this specific question, indicating that things got 
worse. 
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Figure 2 
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When asked, “What has been going well and not well in your role as teacher during 
COVID?”, the strength appeared to be in the technologies that allowed for synchronous 
communication and social presence. On the flip side, the weaknesses mirror other responses, 
specifically dealing with the platform flexibility, the preset curriculum, difficulty with grading 
and attendance policies, workload issues, and the general lack of resources. The lists below show 
responses to the question, ““What has been going well and not well in your role as a teacher 
during COVID-19?” 


Going well: 


e Zoom and other virtual conferencing software 

e Seeing students and connecting with students 

e Many students have become self-directed 

e Collaborating with teachers 

e Teachers sharing tasks: e.g. one teacher creates a video and the other takes another task 
e Lots of resources 

e Having 1:1 computers at home 

e Supportive teaching teams and online teacher groups 


Not going well: 
e Students struggled with motivation and engagement 
e Incomplete work or no work submitted 
e Students are not ready for digital education: struggle with completing assignments 
e Students are overwhelmed and frustrated; do not understand process of e-learning 
e Students and parents lack of resources and tech support while at home 
e Internet disruptions 
e Attendance 
e Honesty and integrity, students are cheating 
e Staying connected: more meetings, email, calls than usual 
e Reduced work speed with higher workload 
e Struggles with instructional design online 
e Preset curriculum is hard to follow 
e Online feedback, grades, and management is a difficult, long process 
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e Lack of expectations & guidance from school/ISBE 
e Completing IEPs 
e Parents did not see e-Learning as school 


Teachers were more confident in transitioning to e-learning during COVID when they were 
already 1:1 and using e-learning technologies when they were allowed to be flexible in how and 
when they could teach, and when schools were fair and clear in their expectations. They also 
appreciated when their administration already had a plan for emergency e-learning situations and 
were consistently communicative, and when parents were responsive in noting their child’s 
participation and completion of work. Teachers’ confidence was greater when they worked 
together to set goals and plan curriculum, when monitoring technology was available, and when 
they actually had more time to complete the additional work that e-learning required of them. 


Technology Tools Used 


Our qualitative data revealed that teachers used a wide variety of digital tools as they transitioned 
to fully online learning when their schools closed. Tools they mentioned ranged from e-learning 
platforms to video conferencing tools, to assessment apps, to educational apps across various 
subjects. About three quarters of teachers who responded mentioned using at least two tools 
besides their school’s e-learning platform to teach students. The most popular was content 
creation tools and video conferencing tools. See several examples of the technology tools for e- 
learning during COVID-19 mentioned by respondents below. Of the wide variety of e-Learning 
platform tools used, Google Classroom was the most common with 53% of respondent’s 
mentioning it (see Figure 3). 


e e-Learning Platform/Learning Management System (LMS): Google Classroom, Class 
Dojo, Canvas, PowerSchool, Haiku, Seesaw, Flipgrid, Padlet 

e Video conferencing tools: Google Meets, Zoom, Microsoft Teams 

e Content creation tools: Google Slides, iPad screen recorder, Seesaw, Google Docs, 
Adobe Creative Suite, YouTube, Microsoft 365, TikTok, Screencastify 

e Assessment: ALEKS, Quizziz, Kahoot, Assignments and quizzes in LMS 

e Education/subject-level apps: Edgenuity, RAZ-Kids, IXL, XtraMath, Freckle, Seesaw, 
Artsonia, Remind App, Mystery Science, Khan Math, Newsela, GoodReader, EdPuzzle, 
Google CS First: game design course 
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Figure 3 
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The e-Learning platforms used appeared to have a noticeable impact on a teacher’s ability to 
promote social, teaching, and cognitive presence. 50% of teachers who responded to the social 
presence question, indicated that the e-Learning platform they were using helped promote social 
presence very well or somewhat well (see Figure 4) Similarly, 49% of teachers who responded to 
the question about e-Learning platforms and Teaching Presence, indicated that the e-Learning 
platform they were using helped promote teaching presence (see Figure 5). However, when it 
came to promoting student learning through reflection and discussion (or ’cognitive presence’), 
most respondents to this question indicated that their e-Learning platform did not do a good job 


helping with that (see Figure 6). 
Figure 4 
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Figure 5 


e-Learning Platforms Promoting Teaching Presence 
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Figure 6 


e-Learning Platforms Promoting Cognitive Presence 
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The data collected also showed variations depending on factors like grade level taught 
(see Figure 7). For example eLearning platforms were seen to help encourage teaching presence 
by a large number of Grade 5-8 teachers (62%) and Grade K-4 teachers (45%). But, eLearning 


platforms were not viewed as favorably by High School teachers. 
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Figure 7 


e-Learning Platforms Promoting Teaching Presence by Grade Level 
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Recommendations from Respondents 


Teachers provided tips and recommendations for both parents and teachers who were new to e- 
learning. Four categories of replies were identified: managing the “new normal,” 
communication, instruction, and technology. 

The first theme was around managing the new normal. Teachers suggested that parents 
and teachers be patient, practical, consistent, simple, and offer incentives for their students. 

The second theme was that of communication. Teachers suggested that reaching out 
should be reciprocal among parents, teachers, and students. They identified different ways to use 
virtual meeting time beyond instruction, including check-ins, Q&A sessions, games, show and 
tell, and guided reading and discussion groups. They also suggested that communications should 
focus not only on academics but also on empathy and physical, social and emotional wellness. 

The third theme of tips for teachers and parents centered around instruction. Tips 
included slowly moving activities from easy to difficult, being creative and experimenting with 
activities, collaborating with other teachers for ideas across subjects, getting regular feedback 
from students, regularly assess students with low-stakes assignments like quizzes, and consider 
using grades and scores even if it’s not required by the state to increase motivation. 

The final theme was that of technology. Teachers advocated for using a combination of 
tools such as e-learning platforms, conferencing software, apps, and email. It would be best if 
these were tools that students, teachers, and parents were already familiar with. They also felt 
that preparing and organizing digital content such as videos and screencasts of lessons was key, 
although time-consuming sometimes. Lastly, they suggested using technology that was fun and 
motivating, such as games, simulations, modeling videos, and activities with scores. 

Teachers were also asked for any suggestions for their school or district administration. 
Their main comments were on the need for planning, policy around grading, technology 
resources such as 1:1 and e-learning platforms, dedicated technology support for students and 
parents, and ongoing professional development. They also appreciated flexibility, 
communication between administration and staff, and more support for converting print 
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resources and guidance around e-resources that were opened up to the public during COVID. A 
more detailed list of these suggestions for administration during COVID-19 is provided below. 


e Puta plan in place and provide specific instructions for teachers, staff, students, parents 

e Grading policy: graded assignments leads to higher motivation and participation 

e Participation standards/accountability need to be established 

e e-Learning platform should be in place 

e Equal access and training for student 1:1 technology 

e Dedicated technology support for students; teachers shouldn’t be tech support 

e Ongoing professional development and teacher community/forums; subject specific 

e Flexibility — students/parents/teachers need a break 

e Consistent communication 

e More synchronous Q&A sessions between administration and staff 

e More support for managing and converting print resources 

e Take advantage of e-resources made free/available by companies and organizations 
during COVID 


Discussion 


The results suggest several issues and opportunities for making decisions about potential next 
steps and future research to support in-service K-12 teachers facing the transition to e-learning 
amid COVID-19. These include the need for further pedagogical support for K-12 teachers in 
integrating e-learning tools and activities, especially on when and how to use synchronous versus 
asynchronous strategies for different subjects and grade levels. Teachers also demonstrate a need 
for clearer guidance on how to develop and implement consistent and reasonable assessment and 
attendance policies in emergency e-learning situations that also support student engagement and 
motivation. The role of technology access, training, and implementation in the PK-12 e-learning 
environment is apparent as well. There were clear differences for teachers in schools that already 
had a 1:1 technology program before the pandemic. 

The results also suggest that there are clear needs for implementing and balancing 
Garrison et al.’s (2000) three constructs of social, cognitive, and teaching presence within the e- 
learning environment. Achieving this balance requires instructional design that incorporates not 
only the presence of the instructor, but also uses technology that can enhance student 
engagement and cognitive presence in synchronous and synchronous activities and 
communications. While teachers stated that e-learning platforms mentioned in this study were 
useful for social presence and teaching presence, they were not as useful for cognitive presence. 
The presences need to be intentionally created as these three elements are foundational to the 
development of deep and meaningful educational experiences in e-learning environments 
(Dunlap, Verma, & Johnson 2016). 


Conclusion 
In addition to additional professional development for teachers and administration about 
pedagogy as well as policy, researchers should conduct a closer analysis for themes around the 


teaching, social, and cognitive presence in e-learning at the PK-12 level. The developers of e- 
learning software and hardware can benefit from understanding the importance of this 
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framework in the development of their products as well. Evaluating the technologies most 
frequently mentioned in this study are a starting point. Follow up surveys with teachers and 
administration would aid in this regard to better understand their next steps pending the decisions 
made concerning e-learning in the new school year. 

While many are beginning to embrace the new normal of the pandemic, others continue 
to believe that we will return to what we were prior to the pandemic. This may or may not be 
true; but we cannot wait to find out. We need to find more ways to replicate the avenues for 
cognitive presence, teaching presence, and social presence not only in our academic online 
classes, but also reflectively in our lives. Using technology effectively, adapting the pedagogical 
approach to teaching online, and opening our minds up to change are a good place to start. 
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Abstract 


Instructional video is a dynamic and important content-delivery tool in online courses. 
However, instructional designers, when providing support to faculty in the design of online 
instruction, may not accurately understand faculty’s expectations and perspectives. This study 
attempted to analyze the perspectives of the designer and those of the faculty to enhance mutual 
understanding between the two groups. From these two perspectives, we attempted to derive the 
best practices concerning the production and employment of instructional videos in online 
courses. Two surveys were administered to collect data for the study, one to the faculty and 
another to the instructional designers at a major eastern state university in the United States. 
Based on the survey results concerning the use of instructional videos in online education, 
faculty and instructional designers share similar views regarding best practices of using videos in 
online courses. However, differences exist in terms of the video styles, and the optimal video 
length. 


Introduction 


Video is a dynamic type of instructional material, as compared with many traditional 
materials. Research has shed light on the important role of videos for effective teaching. Berk 
(2009) discussed key advantages of instructional videos such as inspiring, engaging learners, and 
fostering deeper learning. Sabli¢ et al. (2020) analyzed 39 peer-reviewed articles on video-based 
learning and supported positive effects of online instructional videos. 

The widespread use of instructional videos in online courses brings immense benefits and 
challenges to both instructional designers and instructors in terms of effective incorporation of 
videos in online teaching. Research has indicated that using instructional videos alone in 
teaching does not necessarily improve learning. Zhang et al. (2006) stated that “simply 
incorporating video into e-learning environments may not always be sufficient to improve 
learning” (p. 11). Their study suggested that “interactivity can be a valuable means to improve 
learning effectiveness in an e-learning environment” (p. 11). Similar to Zhang et al.’s findings, 
simply including video lectures does not necessarily produce better learning outcomes (Evans & 
Cordova, 2015). If not well designed, lecture videos may not be superior to still slides and text. 

The above research entails numerous questions concerning the effective use of videos for 
online instruction. This study answered the following research questions: 1) What best practices 
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are used by faculty in instructional video production to engage students? 2) What best practices 
are used by instructional designers in instructional video production to engage students? 3) To 
what extent do faculty use these practices in their online courses? 4) To what extent do 
instructional designers/developers use these practices in their instructional design work 
supporting faculty teaching online? 

With those questions in mind, we designed and administered a survey to a group of 
instructional designers and faculty in different departments of a large state university. By 
exploring and analyzing the views of instructional designers and those of the faculty regarding 
the uses of instructional videos in online courses, we attempted to find answers to the above- 
mentioned research questions. Our findings support the statement that instructional videos in 
online courses, only when effectively designed, developed and employed, can enhance student 
engagement and effectively support students’ achievement of desired learning outcomes. 


Theoretical Frameworks 


Cognitive Load Theory 

Learners process instructional information initially in working memory (Sweller, et al., 
2011) which is then stored semi-permanently in the long-term memory (Atkinson & Shiffrin, 
1968; Tulving, 1972). However, there is a limit to the amount of new information a learner’s 
working memory can process at one time. When information presented to a learner is 
overloaded, the learner’s cognitive capability will be overwhelmed. Consequently, the learner’s 
working memory cannot process the information effectively. The working memory load is 
determined by two types of cognitive loads - the “intrinsic cognitive load” which is imposed by 
the structure of the information and the “extraneous cognitive load” which is imposed by the 
manner in which the information is presented (Sweller, et al., 2011). The extraneous cognitive 
load can cause additional stress on the learner’s processing capability. High extraneous cognitive 
burden would be generated on the mental process if improper use of instructional design is 
employed in course design (Cierniak et al., 2009). When creating instructional videos, faculty 
and video producers can incorporate certain strategies to avoid information overload so that 
learners can process information efficiently. 


Motivation Strategies 

Keller (2009) described a model for designing the motivational aspects of learning 
environments to stimulate and sustain students’ motivation. The model includes four categories - 
attention, relevance, confidence, and satisfaction (ARCS model). Incorporating the elements in 
each category can motivate students to learn the content. When creating instructional videos, 
faculty and video producers can incorporate motivation strategies to encourage student 
engagement in viewing videos. 


Student Engagement 

Student engagement has been defined as investment or commitment (Marks, 2000), 
participation (Kuh, Kinzie, Buckley, Bridges, & Hayek, 2007), or effortful involvement in 
learning (Pekrun & Linnenbrink-Garcia, 2012; Reschly & Christenson, 2012). Video 
engagement is a metric regarding how a viewer watched a video, dropped the video in the middle 
or watched until the end. This type of engagement time is used by both free video providers and 
enterprise providers. However, this engagement metric suffers limitations regarding whether a 
watcher is actively watching the video or merely playing it in the background while multitasking 
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(Guo et al., 2004). As indicated by the study of Guo and others on MOOC videos, shorter videos 
are much more engaging, videos under 3 minutes have the highest engagement level, and the 
median engagement time is at most 6 minutes, regardless of total video length. They provided 
seven recommendations on video lengths, style, etc. for instructors and video producers. The 
recommendations, in summary, include logical chunking of the video, appearance of instructor’s 
face at opportune moments, visual fluency, speaker spontaneity and dynamism, suitability for 
teaching platform, and supporting resources for tutorial videos. 


Methods 


This study examined best practices in the use of instructional videos in online courses 
from both the instructor’s and the instructional designers’ perspectives. Faculty members and 
instructional designers in a state university on the east coast were surveyed in 2020. Data were 
collected and analyzed to investigate best practices, and the differences and similarities among 
views of faculty and instructional designers. Descriptive analysis was utilized to identify best 
practices of instructional videos used by faculty members and instructional designers in online 
courses. 


Research Sites and Participants 

The participants of this study were faculty and instructional designers at a US university 
on the east coast. The university has strong instructional design and technical support to faculty 
in online course design and delivery. Two surveys were designed for the study, one for input 
from the faculty (37 participants) and another for input from the instructional designers (20 
participants). Among the faculty respondents, 62.2% were female; 35.1%, male; and the rest, 
undisclosed. The faculty employment status included full-time (64.9%) and part-time (35.1%). 
All respondents had at least one-year online teaching experience with 22.2% of them having six 
or more years of online teaching experience. Among these faculty participants, 91.4% had used 
instructional videos in their courses. The faculty respondents had taught courses at all levels 
(undergraduate, graduate, and certificate) in varied disciplines (e.g., agricultural sciences, arts 
and humanities, business, engineering, health science, and medicine). 

Among the instructional designers who responded, 56.3% were female, 12.5% male, 
6.2% genderqueer, and 25% undisclosed. Their roles in online course design and development at 
the institution included designing online courses (50%), providing pedagogical support (68.8%), 
providing technical support (75%), developing online courses (25%), assembling courses in 
LMS (62.5%) and producing instructional/promotional videos (6.3%). The majority (81.3%) of 
the designers who responded had at least six years of experience. 


Instrumentation 

Based on the researchers’ experiences teaching online courses and supporting faculty 
members in course development with videos, two instruments were created to gain input from 
faculty members and instructional designers on best practices of using instructional videos in 
online courses. One instrument was designed for faculty and one for instructional designers due 
to the difference in their experiences in using instructional videos in online courses. 

Both instruments included questions to gain their perspectives on using instructional 
videos in online courses, such as the length, type, style, time, number of video clips, strategies 
that motivate students to view videos, and the degree to which instructional videos contribute to 
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student-content interaction. Five questions regarding participants’ demographics were also 
included. 


Data Collection and Analysis 

The study was approved by the university’s IRB committee. The surveys were sent to 
352 faculty members and 20 instructional designers in the university. Thirty-seven and 16 
responses were collected respectively from each group with response rates of 10% for faculty 
and 80% for instructional designers. Data were downloaded and analyzed with Excel for 
descriptive statistics, responses to open-ended questions were coded to reveal themes on 
strategies in using instructional videos to promote student-content interaction. 


Results 


This study endeavored to answer the following central research questions: what best 
practices are used to what extent by faculty and instructional designers in their production and 
employment of instructional videos in online teaching? This section reports results from our 
surveys to answer the above research questions. 

Optimal Video Length. Both faculty and instructional designers responded that shorter 
videos less than 10 minutes work the best. As illustrated in Fig. 1, only 13.50% of faculty 
indicated that the optimal length of videos is more than 15 minutes, while none of the designers 
agreed to this as the optimal video length. The most popular “optimal video length” seems to be 


6 - 10 minutes (as perceived by 45.9% faculty and 68.8% instructional designers). 
Optimal Video Length 


80.00% 
68.80% 
70.00% 
60.00% 
50.00% 45.90% 
40.00% 
29.70% 
30.00% 
18.80% 
20.00% 13.50% 
12.50% 10.80% 
10.00% 
0.00% i 
0.00% 
More than 15 mins Between 10toiSmins Between 6to 10 mins Less than 6 mins 


@Faculty Instructional Designers 


Fig. I Optimal Video Length 


Types of Videos. Faculty reported that they have used varied types of videos to engage 
their students in their online courses. As illustrated in Fig. 2, the most widely used videos are 
voice-over of PPT slides (77.80%), and videos from YouTube or publishers (77.80%). The 
second widely used video is screencasts (38.90%), and the third group is guest speakers 
(27.80%), lecture captures (25%), and talking head (22.20%). Faculty also reported other types 
of videos such as clips from television news, write on blank ppt slides with stylus (mimic 
chalkboard), and videos created by students showcasing their work. Concerning the same 
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question, most instructional designers recommended videos of PPT slides with voice-over (80%) 
and screencasts (80%). They also recommended videos from YouTube or publishers (73.30%), 
guest speakers (73.30%), lecture captures (73.30%), and talking head (60%). Instructional 
designers also recommended lightboard video captures. Instructional designers indicated that the 
type of video used varies by the learning goals. 


90.00% 
80.00% 

77.80% .°0* 77.80% 
60.00% 23 50% 73.30% 
70.00% 
60.00% 
50.00% 46.70% 
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30.00% 
20.10% 
20.00% 
10.00% 
0.00% 
Voice-over Video from Screencasts Guest speakers Lecture I head —__Lightboard Others 
PowerPoint YouTube or captures video captures 
slides publishers 
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Fig. 2 Types of Videos 


Video Styles. As illustrated in Fig. 3, faculty reported they used varied styles of videos in 
their online courses such as formal lectures (69.40%), informal dialogues (38.90%), guest 
interviews (22.20%), and some group conversations (11.10%). Faculty also reported that they 
have created entertaining lectures to engage their students. When asked what type of video styles 
they would recommend to faculty in their online courses, most of the designers recommended 
informal dialogues (50%), and formal lectures (14.30%). Instructional designers also 
recommended using a mix of video styles according to the learning goals and the nature of the 
content. 


When asked about the rationales of their choices of video styles for formal lectures, 
faculty participants offered the following: 

e Students will likely listen to a teacher with a strong online presence. 

e The formal lecture presents content needed to fulfill course objectives. Other formats 
would be harder to do in a succinct way w/o watching multiple sources. 

e Coronavirus didn't leave time to plan creative videos like guest interviews or 
conversations! 

e@ One comment was, “I use this format to get the information across in a quick and easy 
manner. It is somewhat similar to how I would do it in the classroom for this specific 
course.” 

e Another comment was, “I am trying to mimic what they would get in the classroom.” 
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Fig. 3 Types of Video Styles 


When asked about the rationales of their choices of video styles for informal dialogues, 


faculty participants expressed the following rationales: 


Students are likely to listen to a teacher with a strong online presence. 

It's best to be personable when faculty are talking “one on one” with each student. 

These videos are easy to create and often help answer questions students would have 
asked during office hours. Verbal response will be more efficient in conversational 
dialogues. 

Informal dialogue creates more of a classroom, in-person scenario. 

Use of Captions and Transcripts. Captions or transcripts in videos are helpful not only to 


screen-reader users, but also to students for whom English is not their native language. Although 
faculty believed that captions or transcriptions should be included in videos, only 36% of the 
faculty had sometimes, often, or always included captions or transcriptions to their videos, while 
the rest (64%) of them had not. 


Number of Videos in a Module. Faculty also reported that the appropriate number of 


videos included in one learning module should be 1 to 2 (31.40%), 3 (17.10%) and more than 3 
clips (8.60%), as presented in Fig. 5. However, the majority of faculty reported that all depends 
on varied factors such as the length of videos, the nature of the content, and the learning 
objectives. 
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Video Captions or Transcripts 
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Fig. 4 Use of Captions and Transcripts 
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Fig. 5 Number of Videos in a Module 


Strategies to Enhance Student-Content Interaction. As presented in Fig. 6, faculty and 
instructional designers both reported that short instructional videos enhance student to content 
interaction to a great extent (46.80% and 67.70% respectively) or to some extent (51.40% and 
31.30% respectively). 
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Fig. 6 Enhancing Student-Content Interaction 


Strategies to enhance student-content interaction. Faculty also reported that they have 


employed varied strategies in their videos to enhance student to content interaction. These 
strategies include: 


Using humor, visually appealing, enthusiastic in tone, incorporating questions, drawing 
connections to class assignments, providing examples that they can follow along in their 
computers. 

Connecting videos with learning activities such as small group discussions, application 
activities, reflection papers, and Q&As. 

Making video lessons mandatory before coming to Zoom sessions. 

Connecting videos to high-stake assessment activities such as midterms and finals. 
Instructional designers shared following strategies to enhance student to content 


interaction in videos. 


Making them interactive or adding a question set targeting the video info to follow up 
with assessments. 

Embed video quizzes. 

Make sure the video is necessary, make sure the video's purpose/relevance is clear, and 
provide an opportunity for action afterwards. 

To better engage students, keep each video focused around one main idea or topic. 

Flip the class and only discuss the most recent recorded lecture when interacting live. 
Students may not proceed until they have watched the videos, a short knowledge check / 
quiz based on the video (better if graded). 

Faculty reported a number of challenges when creating their own instructional videos. 


The challenges include time, equipment, tech skills, quiet space, and resources such as pictures 
and cool effects. 


Instructional designers also reported their challenges when supporting faculty in creating 


instructional videos. The challenges that designers reported echo those reported by faculty, such 
as time commitment, technical skills, learning curve, motivation to start early, resources, and 
equipment quality. 
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Discussion and Recommendations 


Produce short high-quality instructional videos. The question is how short videos 
work best to motivate students to watch the video completely. Research (Guo, et al 2014) 
indicates videos of six minutes work best. According to their study, the median engagement time 
for videos less than six minutes long was close to 100%. In other words, students tended to 
watch the entire video without dropping out. The views of the faculty and instructional designers 
in our study tended to support this view that shorter videos less than 10 minutes work the best. 

Use the right type of videos. Videos can be created in multiple ways, such as voice-over 
slides, screen-cast, Khan-style drawing tutorials or lightboard videos, talking-head with 
minimum slides, or a mixture of talking heads, relevant visuals, embedded skits, application 
story, ponding questions, pop quiz questions, etc. to make instructional videos interactive. 
Research (Guo, et al 2014) indicated mixing the instructor’s talking head with other visuals and 
materials at opportune times is more engaging than a single video format. The views of the 
faculty and instructional designers in our study also supported this view that varied types of 
videos be used according to the learning goals and the nature of the course content. 

Employ a style that fits your goal. Usually, there are two styles to adopt for producing 
instructional videos: conversational and formal lecture. Research (Meyer, 2008) indicates that 
the use of conversational style rather than formal lecture showed a positive effect on students’ 
learning. The conversational style works better than the formal lecture style as the conversational 
style provides a personal touch on what the instructor presents and makes the student feel 
connected to the instructor and the topic. However, not all instructors are comfortable with the 
conversational style on campus, and not all videos should be produced in a conversational style. 
Decisions should be made according to the goals and the instructor’s teaching style. 

Tie assessments to your videos. Students may take many courses and are involved in 
extracurricular activities simultaneously. If the videos are not tied to their learning assessments, 
it’s likely that some students will not watch those videos. Therefore, it is necessary to create 
online discussions, group work, reflection activities on the short videos and provide clear 
instructions on how students’ learnings from the videos will be assessed. 

Create an active learning activity based on the videos. In addition to tying videos to 
assessments, creating an active learning activity based on the video is another option. For 
example, you can create a “Watch-Think-Write” activity to activate student learning (Serrato, 
2016). Before students watch a video segment, provide several prompts, ask students to watch 
the videos with those questions in mind. Then ask the whole class or small groups to discuss the 
video. The last phase is to ask the students to reflect on what they have watched and answer the 
questions, and then pose new questions to their groups or the entire class. 

Include an appropriate number of videos per module and sequence videos properly 
in a module. Research (Beatty et al, 2019) indicates that student video viewing was lower 
following the initial part of a muti-video sequence for a topic. They suggested frontloading 
important content of a topic at the beginning of a video sequence or subdividing units into 
smaller units to include fewer videos in a sequence. 

Instructional video is a dynamic and important content-delivery tool in online courses. 
This study surveyed faculty and instructional designers at a major state university to understand 
experiences, expectations, and perspectives from both groups. The study intended to improve 
mutual understanding between the two groups and to derive best practices concerning the 
production and deployment of videos in online instructions. 
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One limitation of the current study is the lack of perspectives of students, who are the 
ultimate users of instructional videos and should thus be the ultimate judge of the effectiveness 
of instructional videos. Future studies may need to further investigate students’ perspectives to 
afford a more comprehensive and more useful understanding of the whys and hows of effective 
production and deployment of instructional videos. 
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